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SUMMARY 



Problem 

This study explored methods for providing a common definitional/ 
orientational framework for the Instructional Systems Development (ISD) 
model so that each step in the ISD procedure could be based on a common 
set of integrating and coordinating concepts developed from the first step 
of the ISD procedure. Methods were also examined for integrating the 
various steps of the ISD procedure to allow for more systematic evalua- 
tion of ISD methods and improvement of communications among training 
program developers. The study also investigated the potential of certain 
advanced and novel testing techniques compared with the usual multiple 
choice tests currently being used in Air Force technical training. It was 
felt that certain novel testing techniques could provide for richer and 
more varied evaluative feedback that would aid in course and student eval 
uation. 



Appro ach 

In order to derive the basic job dimensions of the electronic prin- 
ciples and administrative specialist Air Force specialties, two lists of 
critical or frequently performed tasks were compiled from Occupational 
Survey Reports (ORSs) and Plans of Instruction (POIs). Multidimensional 
scaling and hierarchical cluster analysis were then employed to derive the 
job dimensions of the two specialties. The following advanced measure- 
ment techniques were constructed on the basis of the job dimensions de- 
rived from the preceding analyses: sequential, confidence, pictorial ab- 
surdities, pictorial, analogies, cognition of figural systems, partial know- 
ledge, technical words, and Thurstone scaling. These advanced measure- 
ment techniques were administered to samples of electronics principles 
and administrative specialist students. Concurrent validity, reliability, 
cost/benefit, and attitudinal data were collected to evaluate the advanced 
measurement techniques. 



Resu lis 



The results indicated that multidimensional scaling analysis of 
job tasks possesses considerable potential in ISD by providing a com- 
mon framework for application of each step in the ISD procedure. Sev- 
eral of the advanced testing techniques seemed to be potentially more 
useful than the multiple choice tests being used, based upon prediction 
of end of block test scores. The attitudes of the Air Force technical 
training instructors and students enrolled in the two courses were, for 
the most part, favorable toward the advanced measurement techniques. 

Cone lusions 

1. The multidimensional scaling method seemed to possess merit 
as a method for ordering job analytic data and for providing coherency 
within ISD application. 

2. The advanced and novel testing procedures seemed to provide 
an adequate alternative or adjunct to the multiple choice format currently 
being used, 

3. Psychometric, cost/effectiveness, and related properties favor 
ed consideration of the following advanced measures: sequential testing, 
figural systems, confidence testing, technical words, absurdity recogni- 
tion, partial knowledge, and scoring on the basis of the theory of signal 
detection (d'). 
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I. INTRODUCTION 



Technical training within the Air Force is developed through 
a closed loop scheme known as the Instructional System Development 
(ISD) technique. The system is based on a systematic set of steps 
which aim to produce technical training programs which are maximal- 
ly consonant with on-the-job requirements. 

Within the ISD method, one step is concerned with test and eval- 
uation. This test and evaluation aspect aims to develop measurement 
instruments/ methods for determining whether or not the student has at- 
tained formalized criterion objectives. 

The present program possessed two separate but interrelated 
purposes. The first purpose was to demonstrate methods for integrat- 
ing the various steps in the ISD progression. The second purpose was 
to explore and demonstrate the utility of novel test concepts for student 
achievement measurement. 

The ISD Model 

Specifically, the revised ISD (AF Manual 50-2, 1970) model is 
composed of five formal steps: 

1. Analysis of System Requirement s-- ident if ica- 
tion of the tasks to be performed within the overall 
environment of the operational system. 

2 . Definition of Educational or Training Require- 
ments'-determining the tasks that require instruction, 
the level of student proficiency to be developed , and 
the resources needed to conduct the instruction . 

3* Development of Objectives and Tests-'-identz f ica-- 
tion of the behaviors required for successful job per- 
formance and constructing cr iter ion objectives and 
teaching steps, as well as achievement tests. 
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4. Planning, Development , and Val idation of 
Instructions-selecting instructional methods , media , 
and equipment that best satisfy learning objectives , 
determining the sequencing of the instructional ma- 
terial, validation of instructional materials to in- 
sure that all element s of the instructional system 
function effectively in achieving stated objectives . 

5. Conducting and Evaluating Instruct ion-- iden - 
tification of problem areas and corrective actions 
needed in order to satisfy the requirements of the 
operating commands. 



It is possible, however, that if different persons with different 
points of view perform the various ISD steps, a certain amount of loose- 
ness might enter the instructional system. For 'example, the person de- 
veloping the training requirements (step 2) might perceive the job anal- 
ysis (step 1) differently from the original job analyst. And, the person 
developing the objectives (step 3) might perceive the requirements dif- 
ferently from the person who developed them, and so on. The result 
could be the introduction of considerable noise at various system nodes. 
This noise could reverberate and resonate throughout the entire system, 
with the end result that coherency is lost. What is needed is a method 
for ensuring system integrity. Specifically, the present report takes 
the point of view that if a set of integrating and coordinating constructs 
can be developed on the basis of the results of step 1, then these con- 
structs can serve to define and focus each of the succeeding steps. The 
end result would be a common definitional/ orientational framework 
throughout. Moreover, such a common framework would allow more 
systematic evaluation of the ISD methods and, more importantly, fur- 
nish a basis for communication among those involved in the training 
program development. For example, if the fifth step (conducting i^ nd 
evaluating instruction) is conducted in terms of criteria which are dif- 
ferent from the training requirements (step 2), there can be no basis 
for discussion between the two groups performing the separate steps. 
The present report takes the point of view that multidimensional scal- 
ing analysis or cluster analysis of the task identified in step 1 (analy- 
sis of system requirements) can provide the needed common integrat- 
ing core. 
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Mul lidinuMiMioiial Scaling Analysis (MSA) 



The two central problems in MSA are the determination of: (1) 
the minimum dimensionality of a given set of stimuli, and (2) the scale 
value of each stimulus on each of the dimensions. The specific experi- 
mental and computational procedures used have been described in de- 
tail by Torgerson (1952, 1958), Messick (1956a), and Messick and 
Abelson (1956). 

The basic judgment on which the whole structure of MSA rests 
is very simple. In order to obtain estimates of the * 'psychological dis- 
tances" among the various stimuli in the set, most experimenters have 
asked the subjects (judges) to indicate, in some manner, the degree of 
overall similarity between each stimulus pair. The methods for obtain- 
ing and scaling these similarity judgments are generally analogous to 
the classical psychophysical scaling techniques. 

If the obtained scaled values can be taken as measures of the in- 
terstimulus distances in a Euclidean space, the analytical problem then 
becomes the determination of the number of axes in that space and the 
projections of the stimuli on these axes. In these final stages, MSA 
uses factor analysis. As in factor analysis, for example, the pattern 
of scal e values (loadings) of the stimuli (tests) on each dimension (fac- 
tor) presumably enables the experimenter to attach meaning to, and so 
to name, the dimensions. 

There are a number of technical problems involved in MSA, 
such as the choice of method for obtaining the interstimulus distance 
estimates, the choice of spatial model to represent the distances, the 
determination of the constant required to set the distance estimates on 
a ratio scale (Messick & Abelson, 1956), and the decision as to whether 
a transformation of the basic data is required (Helm, Messick, & Tucker, 
1961), Basically, however, MSA involves the steps of: (1) obtaining a 
matrix of interstimulus distances, and (2) determining the dimensional- 
ity of the space containing the stimulus points. 
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MSA also differs from certain other statistical techniques 
like factor analysis in that the results of the MSA grow out of the 
perceptions of the subjects who make the similarity judgments. The 
organization of the field that it produces is, therefore, the structure 
as perceived by these judges. What they consider correlated will be 
included in the emergent dimensions. Since the perceptions of the var- 
ious types of judges may differ, the resulting dimensionality of the 
data may vary across different groups of judges. 

MSA has been demonstrated to constitute a valid method for 
discovery of the underlying job dimensions in the job analytic context. 
For instance, Siegel and Smith (1965) used MSA to order the dimen- 
sions of the job of Civil Defense Director. Also, Schultz and Siegel 
(1962) and Siegel and Schultz (1963) performed a series of multidimen- 
sional scaling analyses in order to determine the job dimensions of 
Naval Aviation Electronics Technicians. 

Hierarchical Clustering Scheme (HCS) 

An alternative to MSA is some form of cluster analysis. Johnson 
(1967) demonstrated an algorithmic technique for grouping variables 
on the basis of empirical measures of similarity-dissimilarity. The 
technique, much like factor analysis, clusters homogeneous variables 
and is referrred to as a hierarchical clustering scheme (HCS). HCS 
has the inherent advantages of being rapid to compute manually, pro- 
grammable for computer calculation, and open to several types of ex- 
tensions or modifications. 

The hierarchical features of the model are such that each sub- 
sequent cluster in an array of clusters is obtained by combining clusters 
at previous levels. Levels are determined by quantitative values which 
can be taken to indicate degrees of "strength. " Eventually, all vari- 
ables are clustered together to form the strongest clustering; the strong- 
est cluster represents the merging of all previous clusters. 

The clustering method produces solutions that are invariant un- 
der monotonic transformations of the data. Johnson states that the 
monotone transformation processes are dependent on the rank order 
of the data and that cluster analysis can be carried out knowing nothing 
of the data but the rank order. The values assigned to the clusterings 
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aj-c (h»l(vfjnin(?d by I'ank ortliri* so Uuil a monolonic! U^anst'ox'mation of 
Iho similarity matrix ti'ansforms ihi) values of the ciustorings, but 
loaves the clusterings invariant. 

Advanced Measurement Techniques 

As stated above, step 3 of the ISD technique involves the con- 
struction of measurement tests. Within the various Air Force tech- 
nical training schools, these measurement tests often take the multiple 
choice format. While multiple choice achievement tests possess a num- 
ber of advantages, there is reason to believe that other paper and pen- 
cil testing methods would provide a more varied and richer student 
evaluative basis. Accordingly, the second purpose of the present work 
involved exploration of some alternate test methods in the Air Force 
technical training context. 

Additionally, the content areas covered by these alternate test 
approaches (called in this report "advanced" or "novel" measurement 
techniques) were drawn from, and based on, the results of the multi- 
dimensional scaling approach. Accordingly, the procedures serve as; 
a test of the utility of the multidimensional scaling procedure for pro- 
viding an integrating thread between the job analysis and the measure- 
ment testing. 

Specific Purposes and Overview of Present Program 

The specific purposes of the present program were to: (1) explore 
the utility of the multidimensional scaling/ cluster analytic approaches as 
a basis for organizing certain aspects of the ISD procedures, and (2) in- 
vestigate the utility of alternate test concepts in the technical training 
context. 

To achieve these goals, two Air Force courses were selected for 
study- -electronics principles and administrative specialist. These two 
courses represent wide differences in job content, training approach, 
and general required student aptitude. The MSA and the cluster analy- 
tic techniques were applied to available job analytic data for the techni- 
cal specialties which graduates of these courses enter. Then, advanced 
measurement instruments were constructed on the basis of the extracted 
dimensions/clusters and the reliability, validity, and cost/ effectiveness 
of these novel instruments were determined. The details of the methods 
involved and thfe results obtained are found in subsequent sections of 
this report. 
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II. MlO'J'llODS 



Development and Administration of Task List 

It is important in multidimensional scaling, as in cluster anal- 
ysis, to consider carefully the variables on which the analysis will be 
carried out, since the end result will reflect only the input data. For 
the present research, interest was centered on the behaviors involved 
in both the administrative specialist and electronics technician Air Force 
specialties. Therefore, two lists of behaviorally oriented job tasks were 
desired which would be inclusive of all the kinds of work performed by 
the men in these ratings, but which would not be so detailed as to require 
an impossibly large number of similarity comparisons or as to make the 
judgmental process unreasonably cumbersome. 

With regard to the administrative specialist, the 35 tasks which 
were most critical or frequently performed by lower level airman ad- 
ministrative specialists were selected from relevant Occupational Survey 
Reports (OSR's), The tasks so selected were; 

1. Type correspondence, directives, or reports 

2. Prepare masters for reproduction 

3. Extract information from files 

4. Process and file correspondence 

5. Post and insert changes in manuals, TO's, regula- 
tions, and similar publications 

6. Operate office copying machines such as xerox, thermo- 
fax, or mimeograph 

7. Prepare drafts of correspondence, directives or reports 

8. Maintain active publication files 

9. Maintain index to forms and publications 

10. Maintain personnel locator files 

11. Prepare and dispatch messages 

12. Establish and maintain suspense files 

13. Coordinate work activities with other sections or agencies 

14. Schedule appointments and conferences 

15. Maintain special order files 

16. Prepare administrative orders 
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17. Requisition publications 

18. Process outgoing mail 

19. Maintain locator records 

20. Distribute forms and publications 

21. Receive, time stamp and route messages 

22. Maintair. inventory of forms and publications 

23. Process official incoming mail for routing 

24. Requisition supplies and equipment 

25. Maintain status boards 

26. Maintain duty rosters 

27. Receive and process requisitions for forms and 
publications 

28. Develop procedures for record maintenance and 
disposition 

29. Prepare briefs of correspondence or reports 

30. Maintain current routing guide and distribution lists 

31. Develop and improve work methods and procedures 

32. Operate key punch machine 

33. Sort and distribute personal mail 

34. Edit and review correspondence and reports 

35. Determine requirements for equipment and supplies 

A somewhat different approach was followed for the electronics 
specialist. The Occupational Survey Reports for electronics specialties 
were considered unsuitable because job experts at Keesler AFB indicated 
that these task listings did not adequately represent the Electronics Prin- 
ciples course. It was indicated that the Electronics Principles Plan of In- 
struction (POI), rather than the Electronics Specialist OSR^s, should be 
used to develop the task list. Accordingly, Applied Psychological Serv- 
ices constructed a 26 -item electronics principles task list from the Elec- 
tronics Principles POI. The following items comprised the electronics 
principles list: 
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1, Knowing the purposes, uses, and application ol com- 
ponents 

2, Using equipment and tools properly and carefully 

3, Cleaning up and maintaining orderly work area 

4, Comparing the effects on circuit functi.on of similar 
components 

5, Describing components 

6, Relating measures taken from components 

7, Identifying electronic components and subclasses of 
components 

8, Performing calculations 

9, Identifying electronic components from schematics 

10, Observing changes in measurement's taken from 
electronic components 

11, Inferring changes in component characteristics as the 
result of measurements 

12, Measuring amplitude, capacitance, resistance, voltage 
etc, , with appropriate test equipments 

13, Knowing the principles of circuit function 

14, Knowing the characteristics of components 

15, Knowing component limitations 

16.. Knowing the use, application, and limits of various 
circuit types 

17, Knowing circuit theory 

18, Identifying atomic components and their action 

19, Knowing the theory of operation of components 

20, Explaining effects on circuit function of electronic 
components 

21, Using oscilloscope 

22, Using miscellaneous test equipment 

23, Troubleshooting from schematics 

24, Reading schematics and identifying parts/ components 
on schematics 

25, Employing safety precautions for self 

26, Employing safety precautions for equipment 
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Similarity Estimates 



As a starting point for analysis, the multidimensional scaling 
model requires an estimate of the psychological distance between each 
pair of stimuli, Messick (1956b) has urged the multidimensional meth- 
od of successive intervals as a simple, efficient, and desirable method 
for obtaining these data. He presents evidence and argues (1956c) that 
it: (1) takes less time than other metnods, such as complete triads, (2) 
can therefore handle more stimuli in a given amount of time, and (3) 
produces equivalent results. 

In the method of successive intervals or, as a special case, the 
method of equal appearing intervals, the distance judgments with regard 
to all possible stimulus pairs are indicated by the judge along a scale 
which is provided for him. For the present work, the stimulus material 
(either electronics or administrative, as appropriate) was presented in 
booklet form. At the top of each page in the booklet, one of the tasks 
was shown. Below it at the left side of the page, the remaining tasks 
were listed in a random order which was varied from one page to anoth- 
er, A scale running from 1 to 11 appeared to the right of each of the 
tasks. The scale pointsl and 2 were described at the top of the page as 
representing a judgment of 'Very similar"; points 3, 4, and 5, as repre- 
senting '^moderately similar''; points 7, 8, and 9, as representing "mod- 
erately different"; and points 10 and 11, as representing "very different, " 
Scale point 6, in the middle of the range, was not described in verbal 
terms, A sample page of the administrative specialist form, called the 
Technical Task Inventory, is shown in Table 1, 

The directions asked the subject to compare each task listed with 
the one shown at the top of the page and then to "indicate by check in the 
appropriate column to the right how similar or different the two tasks 
are, " The complete cover page of the form, including the directions, 
is shown in Table 2, 
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Sample 



Similarity Estimate Page 



tMaintain Index To Forms And Publications 1 



Very Moderatel/ Moderately Very 

Similar Similar Different Different 

12 345 6 789 10 U 



Post and insert changes in manuals ^ TO's^ 

regulations, and similar publications 

Prepare and dispatch messages 

Schedule appointments and conferences, , , , 

Develop procedures for record maintenance 

and disposition _ 

Maintain special order files 

Type correspondence, directives, or reports. 

Requisition publications 

Requisition supplies and equipment 

Establish and maintain suspense files _ 

Develop and improve work methods and pro- 
cedures • _ 

Maintain duty rosters 

Process outgoing mail 

Prepare briefs of correspondence or reports. . 

Maintain current routing guide and distri- 
bution lists _ 

Extract information from files _ 

Prepare drafts of correspondence, directives, 

or reports 

Operate key punch machine. _ 

Maintain personnel locator files _ 

Maintain locator records • _ 

Distribute forms and publications _ 

Maintain status boards _ 

Process official incoming mail for routing. . ^ 

Prepare administrative orders _ 

Receive time stamp and route messages. ... 

Process and file correspondence 

Prepare masters for reproduction 

Receive and process requisitions for forms 

and publications 

Edit and review correspondence and reports. 

Maintain inventory of forms and publications. 

Sort and distribute personal mail _ 

<3 77 

ERIC 



Table 2 

Directions for Similarity Estimates 



Name ^ Today's Date 



TECHNICAL TASK INVENTORY 

The purpose of this Inventory is to compare various tasks perform- 
ed by Administrative Specialists. Each task is shown in a box at the top 
of each page. Below it is a list of other tasks. You should compare each 
task in the list with the one in the box at the top of the list and indicate by 
a check in the appropriate column to the right how similar or different the 
two tasks are. There are no "right" or "wrong'' answers to this inventory; 
your best judgments of similarity are the only "right" answers. 

Before you begin, open the booklet and look over the pages briefly 
to get an idea of what tasks are included. Notice that the pages have dif- 
ferent numbers of tasks listed. Some pages have two or more separate 
lists, each with its own boxed comparison item. 

Start working at the beginning of the booklet. Try to vary your check 
marks so that some appear in all eleven columns. Do not hesitate to use 
the extreme responses numbered 1 and 11, if you feel any comparison de- 
serves one of them. 

EXAMPLE 

Maintain Training Records ! 

'■ ■ - - Very Moderately Moderately Very 

Similsj Similar Different Different 

12 345 6 789 10 11 

Prepare records for controlled mail 

Assign specific work to individuals / 

The first check indicates that the person completing the inventory 
thinks that ''prepare records for controlled mail*' is moderately similar to 
''maintain training records. " 

WHEN YOU HAVE FINISHED, CHECK BACK TO MAKE CERTAIN YOU HAVE 
PLACED A CHECK NEXT TO EACH TASK IN THE LIST ON EVERY PAGE. 
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In addition to tht* random order of tasks on each page, tho 
order of pages in each booklet was randomized. Rach subject, then, 
compli^ted a form with differtnitiy ordert^d pag(^s. This control at- 
tempted to avoid contamination from order, set, and fatigue effects. 

Twenty instructors in the administrative specialist course at 
Keesler AFB completed the administrative specialist questionnaire, 
while 29 electronic instructors completed the electronics principles 
questionnaire. Separate sessions were involved for each group. 

A very brief, general description of the program was given at 
the beginning of each session. The booklets were essentially self-ad- 
ministering. Almost all of the instructors completed the forms within 
two hours. 

The subjects were able to understand their task easily. Most 
of them proceeded without difficulty and with almost no questions. 

The Matrices of Inter stimulus Distances 

For each specialty (administrative or electronics) the scale 
value for each pair of job tasks was taken as the mean of the values 
checked along the similarity scale by the judges in the appropriate 
group. The obtained scale values, or intertask distances, for the 
electronics principles instructors and for the administrative specialist 
instructors are presented in Tables 3 and 4, respectively. 



13 



0) 
I— < 
Si 

CO 



ERIC 



0) 

a 

CO 
0) 

•B 

u 

CO 

c 

•iH 

s 

•a 

< 

a 

CO 
0) 

o 

CO 

Q 

CO 
CtJ 

0) 



13 
c 

•iH 

O 
c 

OS 
0) 



»ri iri . % o O "•' O •* ' •-' O 'J* "J O 

CI o to I" f"* CO Tl © CI 'O lO CI CM 00 O *0 lO (o to 

r-oooof«)0>r-r-r-o»t-r-«oo»oot-cooooo«ocotftoo 
CO ^»Ot-cooococoo&cj>*-'**'^*^<oi^*^*ococo*^^o> 

00O>t*^00O>00t*»<O^«O«PC000O>t-'N«O^Wt- 



o o 

00 to CO 



a»tot*<oo>ao^^cooouoo>ao<ooo^oo<o^i'* 

C4t>»o^cic^oo«^0)^''^^v>(0\oa)^coo^ 
ooa)r-^ooa)ooo&oooocotooooooocor~co«o(o 

uo »niOTj»c^ooc*<»^CJOico^t~tooco6^^ 

000>00«0«0^0><Ot*t-t-»ftt*0>000>"«*^00 

o otf^^i'^uouooooouooououououo 

000>0>eOOt"t"'^'OO^^Oi*'*W^OJWOJ 

00 oot~^ooa)oo(0(oegattcot*oouooooot* 

uo ioo''5o»'>oo>rto*'*oo o^ouoo 
to t^^m^QO(oc)c>iooco(ow*^^o>^ 

CO o>QO^a>o>oocoa>co^tot*tt>oooot- 

0(o^toc*)a>cocoo>co^<Ato<Ac4co 
tooot-t-too>t*c^^o>t*''*t*o>oot- 

O U0oOOOOOOOOi'3^"'O 
t-"^COt-OjOOO>'ftCOt-'^OiC*501'J^ 

<t4*tOtOC*00C0C0O>00COCOC000(O 

cooo^'^o^t-oo^uouotomociu^ 

00^^(00)00 ^^O^OO^OOOJ 

U0o>'^''^'^>'^0000'')00 
OiO^t-OlU^^OOOtOOO^^ 

t^^QOt-AAOOO^QOOOOOtO^ 

ouooov^oooo^ouo 
coCooiAoouoaa^Ob(0<Aoo 

toooo>t*ooo>t-t*ootot*t* 



UO o ' 
to o CO ' 



i Q m o m tr> tn o 

' O O CO * o ^ 



t- OJ uo ^ to 2 ^ ^''■^ 

ouoiiommuoo'Oo'') 
(ot«(oMoo>oa>coco 

cotoc^^oooocooooooo 

I'iCOOMtOt-W^-N 

oioir-toi^aooirtt' 
uouooouooo'o 

OO^fOCNlOOOOOCO 
C^0O0O0OC0O)0O*^ 

o o o >n> o o 

eg to <M ^ O 00 



• to 2 ' 



in o o ^ G ^ 

O O 00 CO 00 

CO 00 00 

uo m o o o 
t« C4 o 00 





«o 


o; 






lO 


to 


•o 


o 




o 


00 


00 




00 


(O 


o 


o 


in 






o 


^ 




(A 


00 


CO 




o 


uo 








CNI 






CO 


00 






uo 
















CO 









— C«J«««'U0t0t-000>O"^WC0^00t0t-000>O^CNC0^U0«0t-000>O'^CSC0^ 
^-^wWi-,»^w^p,^wCgcN|WCgNNNMNWC0C0C0C0C0 



IS 



Multidimensional Scaling Analysis 



The methods of multidimensional scaling analysis have been 
fully described in a number of sources (e,g, , Shepard, et al, , 1972), 
These standard methods were followed in the present analysis. Often, 
in MSA, an additive constant is computed which when added to all the 
relative intertask distances converts them to absolute distances (Messick 
& Abelson, 1956), However, prior MSA studies with job task lists 
(Schultz & Siegel, 1962, 1963; Siegel & Schultz, 1963; Siegel & Smith, 
1965) have indicated that, at least for the job analytic context, the con- 
stant is minimal and the correction exerts no effect on the dimensional- 
ity of the results. 

The similarity estimates, as obtained, were first reversed in 
directionality so that increased similarity was indicated by higher num- 
bers. The resulting scale values v/ere converted to proportions of one 
and the resultant values were placed in similarity ("distance") matrix 
form. Each of the matrices was factored by the method of principle com- 
ponents. The resultant unrotated solutions were then rotated to ortho- 
gonal simple structure as determined by the varimax criterion (Kaiser, 
1958), The factor loadings derived from a MSA must be interpreted 
somewhat differently than those which result from a factor analysis, be- 
cause the data input consist of similarity estimates rather than correla- 
tion coefficients. Each factor loading in the present case can be con- 
sidered to represent the similarity between a task and a particular factor. 

In factor analysis, an item is considered, by convention, to be 
meaningfully associated with a factor if it correlates , 3 or better with 
the factor. With regard to our data, though, a value of , 3 would indi- 
cate moderate dissimilarity with the factor; therefore, a value of , 5 
was adopted as the acceptable criterion for item acceptance into a fac- 
tor or dimension. 

The results of the factor analyses resulted in the identification of 
nine orthogonal factors as representing the 35 administrative specialist 
tasks and four factors for the 26 electronics principles tasks. 

Dimensions of Electronics Principles Tasks 

The stimulus task functions with the highest loadings on the first 
of the four factors extracted from the electronics principles task list 
are presented in Table 5, The task functions in factor 1 are primarily 
concerned with those knowledges concerned with circuit and component 
theory and function. Hence, factor 1 was named "Component and Circuit 
^ Knowledge, " 
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T»,s k Pun t' ( i nti 

Number Task Fu notion l«u a.<l j u a 

17 Knowing circuit theory .848 

13 Knowing the principles of circuit function. . . 823 
20 Explaining effects on circuit function of electronic compon^ts .776 
16 ICnowingtheuse, application, and limits of various circuit types .727 
19 Knowing the theory and operation of components . 672 

1 Knowing the purposes, uses, and application of components . 609 

5 Describing components . 605 

4 Comparing the effects on circuit function of similar components .601 

14 Knowing the characteristics of components . 567 

15 Knowing component limitations . 535 
23 Troubleshooting from schematics .510 



The electronic task functions loadxng heavily on factor 2 are pre- 
sented in Table 6. Three of the four tasks are concerned directly with, 
safety precautions. Accordingly, factor 2 was named "Safety. " 

Table 6 

Stimulus Task Functions Possessing the Highest Loadings 
on Factor 2, Safety 

Task Pu n c t i on 

NuBber Task Function Loading 



25 


Employing safety precautions for self 


.928 


26 


Employing safety precautions for equipment 


. 862 


2 


Using equipment and tools properly and carefully 


. 840 


3 


Cleaning up and maintaining orderly work area 


.725 
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The stimubas task functions with the highest loadings on factor 3 
are described in Table 7. These task functions are mainly concerned 
with the use of test equipment; therefore, this factor was called "Test- 



Table 7 

Stimulus Task Functions Possessing the Highest Loadings 
on Factor 3, Testing 

Task Function 

Nusber Task Function Loading 



22 Using miscellaneous test equipment -.788 

12 Measuring amplitude, capacitance, resistance, 

voltage, etc. , with appropriate test equipments 730 



Two weak factors with similar content were combined and are 
called factor 4, The stimulus task functions with the heaviest loadings 
on the combined factor 4 relate to reading schematics in order to iden- 
tify electronic components. This final electronics factor was named 
"Component Identification. " The items with the highest loadings on fac 
tor 4 are presented in Table 8. 
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Table 8 



Stimulus Task Functions Possessing the Highest Loadings 
on Factor 4, Component Identification 

Task Function 

Nttsber Task Function Loading 

24 Reading Schematics and identifying parts/com- 
ponents on schematics 936 
9 Identifying electronic components from schematics 712 

7 Identifying electronic components and subclasses of 

components . 924 

9 Identifying electronic components from schematics .586 



Dimensions of Administrative Specialist Tasks 

For the administrative specialist, the stimulus task functions, 
loading heavily on factor 1, are presented in Table 9. The task func-- 
tions with the highest loadings on factor 1 seem to represent functions 
of the job that relate to the preparation of documents and other com- 
munications. Accordingly, this factor was named ''Document Prepara- 
tion. " 



19 



Table 9 



Stimulus Task Functions Possessing the Highest Loadings 
on Factor 1, Document Preparation 



T%sk Function 
Nuibe r 



Task Function Loading 



1 Type correspondence, directives or reports . 896 

2 Prepare masters for reproduction .760 
7 Prepare drafts of correspondence, directives or reports . 894 

16 Prepare administrative orders . 768 

29 Prepare briefs of correspondence or reports .785 

11 Prepare and dispatch messages .597 

34 Edit and review correspondence and reports .531 



The stimulus task functions with the highest loadings on factor 2 
represent job behaviors involved in the handling of mail and other com- 
munications. This factor was named "Communications Processing. " 



Table 10 

Stimulus Task Functions Possessing the Highest Loadings 
on Factor 2, Communication Processing 



Task Function 
Kdibe r 


Task Punct 1 on 


Loading 


23 


Process official incoming mail for routing 


. 846 


18 


Process outgoing mail 


. 837 


33 


Sort and distribute personal mail 


.755 


21 


Receive, time stamp, and route messages 


. 803 


11 


Prepare and dispatch messages 


.697 
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The stimulus task functions loading most heavily on factor 3 
are shown in Table 11. The tasks with the highest loadings on fac- 
tor 3 relate to updating and indexing publications. Accordingly, fac- 
tor 3 was named "Publication Maintenance. " All significant loadings 
on this factor were negative. 



Table 11 

Stimulus Task F\inctions Possessing the Highest Loadings 
on Factor 3, Publication Maintenance 

Task Function 

Nufflber Task Function Loading 



8 


Maintain active publication files 


-. 882 


9 


Maintain index to forms and publications 


-.856 


5 


Post and insert change in manuals, TO's, regulations. 






and similar publications 


-.857 



The stimulus task functions with the highest loadings on factor 4 
are presented in Table 12. The tasks loading on factor 4 are concerned 
with record keeping and filing. Accordingly, factor 4 was named "File 
Maintenance, " 



Table 12 

Stimulus Task Functions Possessing the Highest Loadings 
on Factor 4, Fire Maintenance 

Task Function 

Nufflber Task Function Loading 

19 Maintaia locator records . 892 

10 Maintain personnel locator files . 793 
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The administrative specialist stimulus task functions with the 
highest loadings on factor 5 are presented in Table 13. The tasks repre 
scnted by factor 5 relate to requisitioning and processing associated 
with the acquisition of supplies and equipment. Accordingly, this fac- 
tor was named "Supply and Equipment Processing, " 



Table 13 

Stimulus Task Functions Possessing the Highest Loadings 
on Factor 5, Supply and Equipment Processing 

Task Function inflrfinp 
Nuober Task Function Loaaing 

24 Requisition supplies and equipment 896 

35 Determine requirements for equipment and supplies -.851 
27 Receive and process requisitions for forms and 

publications 570 



The stimulus task functions loading most heavily on factor 6 are 
presented in Table 14. The tasks with the highest loadings on factor 6 
use action words which relate to filing in the active sense, i. e. , ''proc- 
ess, " "extract, " and "maintain. " This factor, like factor 3, contained 
no tasks with significant positive loadings. Factor 6 was named "Filing. " 



Table 14 



Stimulus Task Functions Possessing the Highest Loadings 
on Factor 6, Filing 



Task Function 

Number Task Function Loading 

3 Extract information from files . 822 

4 Process and file correspondence . 590 
15 Maintain special order files . 616 
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The stimulus tasks with the heaviest loadings on factor 7 are' 
presented in Table 15. The tasks loading on factor 7 refer to the de- 
velopment of clerical methods and procedures. Hence, factor 7 was 
named "Job Structuring and Development. 



Table 15 

Stimulus Task Functions Possessing the Highest Loadings 
on Factor 7, Job Structuring and Development 

Task Function 

Number Task Function Loading 

28 Develop procedures for record maintenance and 

disposition . 874 

31 Develop and improve work methods and procedures . 763 



The stimulus tasks with the highest loadings on factor 8 are pre- 
sented in Table 16. This factor is mainly concerned with the main- 
tenance of lists of various kinds. Accordingly, factor 8. was named 
"List Maintenance. " 



Table 16 

Stimulus Task Functions Possessing the Highest Loadings 
on Factor 8, List Maintenance 

Task Function 

Number Task Function Loading 

25 Maintain status boards . 806 

30 Maintain current routing guide and distribution lists . 513 
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Finally, factor 9 relates to the operation of office machinery 
and was called "Machine Operation. " The tasks with the highest load- 
ings on factor 9 are presented in Table 17, 



Table 17 



Stimulus Task Functions Possessing the Highest Loadings 
on Factor 9, Machine Operation 



Task Function 
Nuabe r 



Task Function 



Loading 



Operate office copying machines such as xerox, 

thermofax, or mimeograph 
Operate keypunch machine 



. 879 
. 805 



Cluster Analysis 

As a check on the factor analysis and in order to explore the merit 
of cluster analysis for job dimensionality determination, a hierarchical 
cluster analysis was performed on the similarity estimates for the separ- 
ate groups. Cluster analysis is a method which is ideally suited to the re- 
duction of intertask distance matrices. No special assumptions are re- 
quired for use of the method, and it can be applied i:o almost any kind of 
distance or correlational data. It has the added aUvantage of simplicity 
and ease of interpretation, A matrix reduction can easily be accomplished 
by hand if the variables number 25 or less. If large matrices are involved, 
a computer program is available which requires only seconds of computer 
time to arrive at a solution. The results of cluster analysis automatically 
produce "simple structure, The method is totally insensitive to data trans- 
formation, a fact which avoids the additive constant derivation. 

The method of Johnson (1967), described previously, was used in 
the present analysis. Table 18 presents the cluster analytic results for 
the electronic principles tasks. The dotted line across Table 18 is the cut- 
off value for acceptance of an item into a cluster. Each stalk or bar in the 
histogram, except for these connecting clusters, represents an item. The 
item numbers are written across the top of the histogram and the clustc^rs 
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Perfonlng Calculations 

Inferring Changes in Coiponent 
Characteristics as the Result 
or Me asu r eten t s 

Using Oscilloscope 

Using Miscellaneous Test £quii/*enc 

Measuring Voltage, Aiplitude, 
Capacitance, Resistance, etc., with 
Appropriate Test Equipments 

Knowing the Theory and Operation of 
Coipon en t s 

Explaining Effects on Cjircuit 
Function of Electronic Components 

Troubleshooting froi Scheiatics 

Knowing the Use, Application, and 
Llalts of Various Circuit Types 

Knowing Circuit Theory 

Knowing the Principles of Circuit 
. run c 1 1 on 

Coiparing the Effects on Circuit 
Function of Siiilar Components 

Observing Changes in Measurements 
Taken froi Electronic Coiponents 

gel at Ing^Measu res Taken froi 
Coipon e n 1 3 

Identifying Electronic Coiponents 
and Subclasses of Components 

Reading Schematics and Identifying 
Parts/ Components on Schematics 

Identifying Electronic Components 
froi Sch ema ti cs 

Knowing Component Limitations 

Knowing tlte Characteristics of 
Coipon en t s 

Knowing the Pu r po Ses , U s e s , an d 

Application of Coiponents 

Describing Coiponents 



Cleaning and Maintaining Orderly 
Work Area 

Eiploying Safety Precautions for Sri f 

Eiploying Safety Precautions for 
Equipient 

Using Equipient and Tools Properly 
ftnd Carefully 
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are included at the bottom. The cluster values are shown at the sides 
of the histogram. Table 19 presents the name given to each cluster 
shown in Table 18, 





Table 19 




Cluster Names for the Electronics Principles 




Clusters Analysis 


Cluster 




Nu iber 


Cluster Haie 


1 


Safety 


2 


Component Characteristics 


3 


Component Identification and Measurement 


4 


Component and Circuit Characteristics 


5 


Electronic Analysis 



Similarly, Tables 20 and 21 present the cluster analytic results 
and cluster names for the administrative specialist questionnaire. For 
both cluster analyses, the item cutoff point was set as close to , 5 as pos- 
sible, because items that cluster below , 5 are more dissimilar than sim- 
ilar to each other, while items that cluster above , 5 are more similar 
than dissimilar to each other. 
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Table 21 



Cluster Names for the Administrative Specialist 





Clusters Analysis 


Cius t er 




Number 


Cluster Name 


1 


Document Preparation 


2 


Machine Operation 


3 


Job Structuring 


4 


Communication Processing 


5 


Filing 


6 


File Maintenance 


7 


Board Maintenance 


8 


Development of Clerical Methods and Procedures 


9 


Form and Publication Processing 


10 


Publication Maintenance 


11 


Supply and Equipment Maintenance 



Algorithmic Integration— Electronic Principles 

Since the multidimensional scaling and the cluster analytic al- 
gorithms are based on entirely different suppositions, any similarity of 
transmethod results is evidence for the robustness of the current solu- 
tions and for the underlying dimensionality of the task data. 

The results of the cluster and the factor analytic algorithms, 
when applied to the electronic principles interstimulus distance matrix, 
were not as strikingly similar as the results derived from the adminis- 
trative specialist interstimulus distance matrix, described below. 

Comparison of the electronic principles results across methods 
revealed that clusters 2 and 4 are totally contained in factor 1. Apparent- 
ly, the cluster analysis, in this case, differentiated to a greater degree 
and separated component characteristics from circuit characteristics. 
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The stimuli that loaded the highest on factor 3 (Safety) are ex- 
actly the same as the stimuli in cluster 1 (Safety), On the pther hand, 
only two of the items in cluster 5 (Electronic Analysis) loaded on fac- 
tor 3 (Testing). 

Finally, the three items in factor 4 (Component Identification) 
are also members of cluster 3 (Component Identification and Measure- 
ment), 



Algorithmic Integration — Administrative Specialist 

The similarity of results across both techniques is quite striking 
for the administrative specialist task data. 

The five items of cluster 1 (Document Preparation) are also con- 
tained in factor 1 (Document Preparation), We note that the two items 
(item 11 and item 34) that f£Liled to cluster in cluster 1 had the lowest 
factor loadings on that factor (, 597 and ,531 respectively). The five 
items that are common to both algorithms, though, had loadings of , 760 
or higher. The items making up factors 2, 3, 6, 7, and 9 are exactly 
the same as the items in clusters 4, 10, 5, 8, and 2 respectively. This 
means that almost perfect content congruency was exhibited for six of the 
nine factors. 

Two items in factor 4 (File Maintenance) were the same as two 
of the three items in cluster 6 (File Maintenance), The third item in 
cluster 6 (item 12) had a cluster value of , 54, while the two common 
items clustered at , 78, As with factor 1, it seems as though those items 
that failed to fall into the same cluster/factor had the lowest loading value 
with that cluster or factor. 

Similar results were obtained for factor 5 and cluster 11, Two 
items in cluster 11 (Supply and Equipment Maintenance) were the same 
as two of the three items in factor 5 (Supply and Equipment Maintenance). 
The one item in factor 5 (item 27) which did not cluster with cluster 11 
had a loading of -,570 on factor 5, 
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Finally, factor 8 seems to be the only factor which did not cor- 
respond to any cluster. The two items in factor 8 (items 25 and 30) fell 
into different clusters. Also, clusters 3, 7, and 9 had no correspond- 
ing factors. 

Accordingly, it seems that the cluster analytic solution produced 
greater differentiation than the factor analytic approach. However, it 
seems that the cluster analytic approach may have overfragmented the 
tasks to produce a result which would be obtained from overfactoring. 
However, these results warrant the conclusion that both solutions yield- 
ed highly similar structure for the administrative specialist interstimu- 
lus distance matrix. 

These results lead to the conclusion that, in the task taxo- 
nomic context, the differences between the results yielded by the two 
methods seem to be that cluster analysis may tend to produce more 
than one cluster for each analytically derived factor. Nevertheless, 
the results from both methods are similar enough to support the con- 
tention that a suitable descriptive structure can be obtained from either 
technique. 

Basis for Test Selection 

As stated earlier, for the tasks performed by the administrative 
specialist and the electronics technician in the Air Force, a multidimen- 
sional scaling analysis was performed. Both factor analytic and hierar- 
chical cluster analytic procedures were applied to yield the basic organi- 
zational structure. The results of the application of both analytic tech- 
niques seemed largely congruent within both specialties. However, the fac - 
tor analytic results were adopted for the advanced test methods develop- 
mental aspects of the present work. 

A separate analysis was performed to yield a list of the types of 
advanced measurement techniques which seemed of value in the Air Force. 
This analysis was largely drawn on a previously developed report (Siegel, 
Bergman, & Sellman, 1972). 
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This U»st lisl was Ihtni cMnployt^d as om) axis o( a malj-ix, wluU' 
Ihe racloriul stx*uctui o ol' the task lisl was omployod as the second 
axis. Koi- each test typo, a factor assignment was made. The results 
fox- electronics are presented in Table 22, (The safety factor was elimi- 
nated from Table 22 inasmuch as students enrolled in Block IV of the 
Electronic Principles course, the course Block from which students 
were subsequently drawn for test tryout purposes, are minimally re- 
sponsible for knowledge of the various safety procedures.) As indicated 
in Table 22, it was our goal to measure each factor (up to the comple- 
tion of Block IV) by a separate test type. Accordingly, there was a con- 
fidence testing procedure applied to measurement of the component cir- 
cuit characteristics course factor, a pictorial test of the component iden- 
tification factor, etc. The need achievement test was not factorially re- 
lated and the psychophysiological reaction was obtained in conjunction 
with the confidence test administration. Scoring in accordance with the- 
ory of signal detection took place in conjunction with the confidence test. 



Table 22 



Test Type for Testing Each Factor in 
Electronics Principles Course 



Course Factor 



Coiponent/Circult Coiponent 
Test Type Chtrtctcrlstlct Testing Identlflcttion 

Sequential Testing / 
Confidence Testing / 

Pictorial Testing /3 
Cognition of Figural Systems /(symbolic) 
Analogies / 
Signal Detection^ / 

Need Achievement^ 
Psychophysiological Arousal^' ^ 

^ Administered in conjunction with confidefnce testing 
^ Not factorially related 

^ Pictorial schematic reading and pictorial absurdities 
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The same logic was applied to the administrative specialist, 
and the results are presented in Table 23. The various advanced 
^techniques are all applied to the same work factor in this case be- 
cause Block II of the course, the block selected as a basis for the 
current work, largely involves learning typewriting within the docu- 
ment preparation context. 

Electronics Tests 

The electronics advanced measurement techniques included: 
(a) sequential testing, (b) confidence testing, (c) pictorial absurdities, 
(d) pictorial schematic reading, (e) configuration of f^gural systems, (f) 
analogies, (g) signal detection, (h) a measure of need achievement, and 

(1) a measure of psychophysiological arousal. 

The development and use of each of these testing techniques (ex- 
cept need achievement and psychophysiological arousal) is described in 
detail in the following paragraphs. The need achievement and psycho- 
physiological arousal methods, which were also employed for the ad- 
ministrative testing, are discussed at the conclusion of the description 
of the various advanced measurement techniques. 

Sequential Testing 

Sequential testing involves a branching sequence in which the 
student receives items at a difficulty level that is appropriate for his 
level of accomplishment. If a student fails to reach criteria on his 
first block of test items, he is routed to an easier block of items. If, 
on the other hand, he exceeds criterion on the first block of items, he 
is routed to a more difficult block of items. This procedure has the 
following advantages; (1) it decreases test time (especially for students 
at the extremes of the distribution because they can be routed quickly), 

(2) it increases reliability, and (3) it increases student motivation be- 
cause the student is not forced to take and guess at more difficult items. 

Construction of sequential tests requires determination of item 
difficulty. Since the items within the tests currently employed in the 
electronic principles Block IV have been extensively item analyzed, 
these items were used as the basis of the sequential test. 
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Five nonoverlapping tests of 12 multiple choice items each 
were selected from the 200 items used in Block IV of the electronic 
principles course. The sequential subtest number, difficulty range 
(the range of difficulty of the various Block IV items selected for in- 
clusion in a particular sequential subtest), and mean difficulty value 
(the proportion of persons in an Air Force pool of electronics stu- 
dents responding correctly to the items) for the five, five minute 
tests developed from these items are presented in Table 24. 



Table 24 

Sequential Number, Difficulty Range, and Mean Difficulty 
Value for Each Sequential Subtest 



Subtest Difficulty Range Mean Difficulty 



1 


.89 - 


. 95 


.92 


2 


.77 - 


.83 


. 80 


3 


.65 - 


.71 


. 68 


4 


.53 - 


.59 


.56 


5 


.41 - 


.47 


.45 



Sample items from sequential test 3 are shown in Figure 1. In 
order to determine if the adjacent sequential tests were significantly 
different from each other, an analysis of variance was performed. 
This analysis compares test 1 results with test 2 results, test 2 re- 
sults with test 3 results, etc. The total N in the analysis was 60, the 
number of items making up the sequential test battery, because the in- 
tent of the analysis was to compare difficulty values across subtests. 
The results of this analysis are summarized in Table 25. 
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1. FREQUENCY DISTORTION CAN BE CAUSED BY 

a, linear operation 

b, non-linear operation 

c, removing bypass capacitors 

d, reactive components 

2. POWER DISSIPATION IN A TRANSISTOR CAUSES AN INCREASE IN 

a, voltage gain 

b, junction resistance 

c, junction temperature 

d, the forward current transfer ratio 

3. SELECT THE SCHEMATIC DIAGRAM OF A SHUNT NEGATIVE LIMIT ER 




4. THE PLACEMENT OF CI MAKES THE FILTER A 



a, capacitor input L type 

b, capacitance type 

c. L type RC 

d. L type LC 



cm 




Figure 1. Sample items from sequential test 3. 
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Table 25 



Summary of Analysis of Variance Comparing Adjacent 
Sequential Subtests 



Source 


SS 


df 


MS 


F 


Between Groups 


62. 861.42 


4 






Comparison: 










1 with 2 


73.35 


1 


73.35 


21.51* 


2 with 3 


70. 36 


1 


70.36 


20. 63* 


3 with 4 


71.86 


1 


71.86 


21. 07* 


4-with 5 


65. 87 


1 


65. 87 


19.32* 


Error (Within Groups) 


187.58 


55 


3.41 




Total 


63.049.00 


59 







* p < .001 



The results indicated that the difficulty levels of adjacent tests 
are significantly different from each other. 

The sequence of administration of the sequential subtests is shown 
in Figure 2. Subtest 3 was administered first. Those students who scored 
above the criterion (six or seven items correct) on this subtest were 
routed to one of the more difficult subtests. The individuals who scored 
below criterion on subtest 3 were routed to one of the easier subtests. Fi- 
nally, students whose scores met the criterion were assigned a hierarchi- 
cal level of three and their testing was culminated. On the second day of 
testing, the remaining individuals were tested on either sequential sub- 
test 1, 2, 4, or 5, depending on their score on subtest 3. Each subject 
who met the criterion on their assigned subtest stopped testing, while 
those subjects who surpassed or fell below the criterion were assigned 
another subtest on the last day of testing. 

Confidence Test 

Confidence testing represents a student response scheme for al- 
lowing more adequate evaluation of student knowledge than traditional 
testing methods. In confidence testing, a student can maximize his ex- 
pected score by reflecting his degree of belief or probability that a spe- 
cific response choice is correct. In the traditional procedure, using a 
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true-false test as an illustration, the student implicitly assigns a differ- 
ent probability for each response depending on his state of knowledge. 
If the student perceives the probability of "true" to be greater than .50, 
he should choose "true. " If he perceives the probability to be less than 
,50, he can choose either response. Generally, a student with poor 
knowledge (p = ,51) will get the same score (if correct) as the person 
with good knowledge (p = , 90); therefore, the choice situation loses data 
about the student's knowledge. 

The "pick one" method of confidence testing was used in the pres- 
ent investigation (Echternacht, Sellman, Boldt & Young, 1971), In the 
pick one method, the student first picks the answer he thinks is correct 
from among various multiple choice alternatives. He then assigns a 
probability value indicating his confidence. This probability value is 
then converted to an item score using special tables. (See AFHRL-TK 
71-33 and AFHRL-TR-71-32 J In the tables, scoring is differentially 
weighted so that both item correctness and student confidence are re- 
flected. Assume the student to assign a .7 probability to his answer. 
The tables indicate a score of . 84 if his answer is correct and 76 if 
his answer is wrong. 

The main advantages of confidence testing lie in Ihe more thor- 
ough assessment of student knowledge that it allows, in the virtual elimi- 
nation of chance exerting an overriding effect on final test score, and in 
the increase in the student's perception of fairness that it provides. In 
addition, confidence scores can serve a training feedback purpose. If 
the situation arises in which the confidence scores of a group of students 
are very low, even though the absolute scores are high, an indication 
of the need for additional training would be evidenced. 

The disadvantages of confidence testing are the scoring difficulty 
and increased time requirements for test scoring. However, those dis- 
advantages are probably outweighed by the advantages. Moreover, com- 
puter scoring can be employed to ease the scoring burden. 

As with the sequential test, the 30 items used in the confidence 
test were selected from the 200 items customarily used in the Block IV 
electronic principles tests. The items delected were representative of 
the total difficulty range (, 40 - , 99) and no item was included in both 
the sequential and confidence tests. 

Characteristically, students have difficulty grasping the notion 
of assigning a confidence estimate to an answer. The formal test direc- 
tions employed for the confidence testing are presented in Figure 3 and 
sample items are shown in Figure 4, 

38 
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ELECTRONIC PRINCIPLES EXAMINATION 



COMPONENT AND CIRCUIT CHARACTERISTICS 
DIRECTIONS 

This is a test of your knowledge of component and circuit character- 
istics. To answer each question, first, pick the answer you think is correct 
from among the multiple -choice alternatives. Then, assign a probability value 
indicating your confidence in that answer. The probability values you can use 
range from . 25 to 1. 00. If you assign a probability value of . 25 to your an- 
swer, this indicates that you have selected randomly (guessed completely) and 
that you are only 25% sure that your answer is correct. Alternatively, if you 
assign a probability value of 1. 00 to your answer, this indicates that you are 
100% certain that your answer is right. If you assign a high probability value 
to an answer and the answer is correct, you will receive a higher score for 
that item than if you had assigned a low probability value for that answer 
(even though the answer is correct). On the other hand, if you assign a high 
probability value to an answer and the answer is wrong, a larger amount will 
be subtracted from your score than if you had assigned a low probability value 
to that wrong answer. In other words, you can get the greatest score by as- 
signing high confidence estimates to those items you feel certain are correct 
and low confidence estimates to those items which you are uncertain about. 

Examples 

1. Which of the following is the best conductor of electricity? 

a. steel 

b. iron 
(cj copper 

d. rubber Confidence Level 

2. Which of the following have a positive charge? 

neutron 

b. electron 

c. neutrino 

d. proton 

In the first example, the student was quite confident that his answer 
(c) was correct; therefore, he assigned a high probability value to that answer. 
In the second example, the student was somewhat uncertain of his choice; ac- 
cordingly, he assigned a low confidence estimate to his answer (a). This was 
a ^jood strategy, since the first answer was correct and the second answer was 
introrrect. If the student had reversed his confidence estimates, his score for 
th<* two items would have been much lower. 

You will have 60 minutes to complete this test. The examiner will tell 
yoj when to begin. 

Figure 3. Confidence test directions. 



Confidence Level 



14. GIVEN: 



1c 



4 ma 

3ma 
2ma 
Ima 











4m(3 










3 ma 










2ma 










Tma 










lc = 0 



5V lOV 15V 20V 
Vc 

COMPUTE ALPHA, USING = 10 V. 

a. .5 c. 1.2 

b. .9 d. 1.6 



Confidence Level 



15. IN THE CIRCUIT SHOWN, THE OUTPUT VOLTAGE HAS DECREASED; 
THE CAUSE IS: 



a. R4 open 

b. Ql open 

c. CRl open 

d. R2 open 



J1 O 




OJ3 



20VDC 



OJ4 



Confidence Level 
[''igurc 4. Sample items from confidence tost. 
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Pictorial Absurdities 

In the electronic principles pictorial absurdities test, the student 
was required to examine nine circuit diagrams. In each diagram (Fig- 
ure 5), some facet was depicted incorrectly. The examinee's task was 
to state what is wrong in the diagram. 



Absurdity items are particularly advantageous with persons who 
are handicapped in their ability to read. Additionally, absurdity items 
test the student's general level of alertness, his ability to recognize 
typical circuits, his knowledge of the circuit's characteristic function, 
his ability to concentrate, and his ability to separate important from ir- 
relevant situational aspects. There is a difference in the electronic ma- 
turity of a student who answers correctly but gives a trivial answer as 
compared with the student who recognizes a major fault. Students who 
can recognize absurdities also are believed predisposed to the ability to 
troubleshoot electronic equipment adequately. This type of item possesses 
high face validity and, accordingly, is more acceptable than the usual mul- 
tiple choice type of item in which language comprehension is heavily in- 
volved. The examinee instructions for the test are shown in Figure 6. 




ANSWER^ 



Figure 5. Sample item from pictorial absurdities test. 
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ELECTRONIC PRINCIPLES EXAMINATION 



PICTORIAL ABSURDITIES 
DIRECTIONS 

This is a test of your ability to detect what is wrong with a picture or diagram. 
You will be presented with a series of nine pictures or diagrams and your task 
is to determine what is wrong with each. There is only one thing wrong in each 
picture. If more than one thing looks wrong, select the one Which is most wrong. 



Example 




Answer 



The above example shows a man wearing wet clothes entering the lab. A 
correct answer for this picture is "Don't be wet when entering or working in lab. " 

You will be given two points credit for each correct answer. One point will 
be subtracted from your score for each incorrect answer. You will receive no 
points for unanswered questions. You will be allowed 15 minutes to complete 
this test. Remember, there is only one thing wrong in each picture. 

R J C Figure 6. Instructions for absurdities. 



Pictorial Test (Schematic Reading/Component Identification) 



To test the component identification factor through pictorial 
methods, a test of the examinee's ability to identify components in a 
diagram was employed. This type of item also possesses considerable 
congruency with the on-the-job aspects of the electronics technician's 
work. 

Each item in the test consists of a diagram which is followed by 
one or more questions. First, the student examined each diagram. Af- 
ter he examined a diagram, he answered questions about the diagram. 
Fifteen minutes were allowed for completion of this test. Figure 7 
presents a sample item from the pictorial schematic reading test. 



Vcc 
Q 




NAME THE COMPONENTS THAT ARE USED 
FOR TEMPERATURE STABILIZATION. 



Figure 7. Sample item from the pictorial schematic reading test. 
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Cognition of Figural Systems 



It seemed wise in any exploratory test development progi^am 
to involve some pure factor approach to intellective function. Tho 
cognition of figural systems test is based on one aspect of the Guilford 
Structure-of-Iniellect model. This model of intellective function has 
had extensive development and verification (Guilford & Hoepfner, 1971), 
The ability is loaded on the apprehension of spatial arrangements of 
items in one's psychological field. As employed here, the analogy 
might be drawn between the incidental development of "cognitive maps'* 
(as suggested by Tolman ir. the maze learning situation) and the develop- 
ment on the part of the proficient electronics technician of "cognitive 
.maps" of the equipment on which he works. The conjecture behind this 
test was that the better technician possesses a higher level of 'Visual in- 
sight, " as the result of his exposure to equipment, than the poorer tech- 
nician with the same experience. 

In the cognition of figural systems test, the subject was presented 
with a series of component board layouts for electronic circuits. Each of 
the component board layouts had several circuit connections missing. Each 
circuit was verbally described at the top of each layout. The subject^s 
task, then, was to draw connecting lines to form th3 indicated circuit. 
The student was allowed 30 minutes to complete this test. A sample item 
from the .cognition of figural systems test, along with the correct re- 
sponse, is shown in Figures 8 and 9. The student directions for the cog- 
nition of figural systems test are shown in Figure 10. 

The other advantages of this technique include its freedom from 
literacy requirements and its inherent interest to the examinee. In ad- 
dition, the test involves a considerable deductive component. 
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COMMON EMITTER AMPLIFIER 
INPUT 



-Vcc 



<D — le- 

-(3) VSAr- 

(5) ^AAr- 




©— Q — ^ 



1^ 



Figure 8. Sample item from the cognition of figural 
systems test. 



COMMON EMITTER AMPLIFIER 
INPUT, 




Figure 9. Correctly completed sample item from 
cognition of figural systems test. 
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ELECTRONIC PRINCIPLES EXAMINATION 
FIGURAL SYSTEMS 
DIRECTIONS 

This is a test of your ability to properly connect the various com- 
ponents and parts of a circuit. You will be presented with a series of 
component board layouts for electronic circuits. Several connections 
are missing in the layouts. The circuit is verbally described at the top 
of each layout. You are to complete the connections to form the indicated 
circuit. Use all components in the layout. 

EXAMPLE 

Half wave rectifier with capacitive filter 




Complete the connections to form the indicated circuit in the above sample 
layout. After you have completed the circuit, turn to the next page where 
you will find the correct answer for this example. 

Figure 10. Instructions for cognition of figural systems test. 
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Figure 10. Instructions for cognition of figural systems test (cont. 

The student completing the example drew in the connections be- 
tween 3-11, 5-7, and 8-10-12. 

You will receive one point for each correct connection. One 
point will be subtracted from your score for each incorrect or missing 
connection. 

You will be allowed 30 minutes to complete this test. 
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The testing factor (factor 3) v;as also measured through an anal- 
ogies test. In the electronic analogies test, the examinee was given a 
series of electronic analogies with one of the four members of the set 
missing. The examinee's task was to complete each analogy by select- 
ing the correct item from a list of four items. The examinees were al- 
lowed 15 minutes to complete this test, A sample electronic analogies 
item is presented in Figure 11, 

Analogies, as here employed, not only test knowledge (the ex- 
aminee must recognize and understand the function and nuances of the 
circuits shown), but they also test the sophistication of his electronic 
thinking. To succeed on the analogies test, the examinee must be able 
to analyze circuits and to think logically. Deductive ability, also be- 
lieved to be important to advanced electronic troubleshooting, is also 
involved in management of analogy items. Moreover, this type of test 
demands concentration and the ability to consider simultaneously several 
interrelated facts. These are also believed important to on-the-job func- 
tion as an electronics technician. In addition, the electronic analogies 
test is unencumbered by literacy requirements. 
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Figure 11. Sample electronic analogy items. 



Signal Detection 



Within the recent past and starting about 1954, the theory of sig- 
nal detectability has emerged. Such theory possesses roots in both de- 
cision theory and in electrical engineering (with reference to ideal sen- 
sory devices), Swets and Tanner (e, g. , Swets, Tanner, & Birdsall, 
1964) generally (and, to our knowledge, first) have taken the lead in ap- 
plying this approach to psychological problems. The novel aspect of 
theory of signal detectability, as employed by psychologists in reference 
to the human organism, is its emphasis on the sensitivity of the human, 
as separate from his role as a decision maker. Thus, persons concerned 
with theory of signal detectability suggest that a sensory threshold per se 
is a will-of-the-wisp and that we should really be concerned with response 
thresholds. The classical problem and approach, in the theory of signal 
detectability, are rooted in the detection of a signal in noise. The basic 
question is, " Assuming a knowledge of the conditional probabilities of 
the signal and the noise distributions, if the observer is presented with a 
stimulus, what is the 'best* rule for him to follow in deciding whether 
the stimulus is signal or noise? " The set of possible outcomes from 
such a trial is shown in Figure 12, where 'V^s" or "no" refer to the 



Response of Subject 
Yes No 



Stimulus Condition 



signal 



noise 



Figure 12, Alternate outcomes from two choice experiment. 
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responses of the observer and where **signal" and ^'noise*' refer to 
wh(ith(jr signal or noise was presented. If the observer says "y^^" 
when a signal is presented (cell 1), we refei* to his response as a '*hit. " 
If he replies ''no*' when a signal is present, we have a "miss'* (cell 2 
of Figure 12), and if he replies ^'yes'* when noise is present (cell 3 of 
Figure 12), a ^'false alarm*' situation is indicated. Finally, if the ob- 
server replies "no" when noise is present, a correct rejection is indi- 
cated. For a large number of goals (and ignoring payoff considera- 
tions, i. e. , the cost of a "false alarm" or of a "miss" are the same 
as the gain from a "hit" or from a correct rejection), the best strategy 
for the observer to follow is governed by the likelihood ratio. The 
likelihood ratio is the ratio of the conditional probability of making a 
correct response given signal [p(x|s)] to the conditional probability of 
a correct response given noise: 



p(x 


s) 


p(x 


n) 



Likelihood ratio 



This ratio represents the best place for the observer to set his response 
threshold or criterion value. As the observer's criterion value deviates 
above or below the optimum, his "hit" or "false alarm" rate will vary. 

If the "hit" and "miss" rate are plotted against each other, we 
have a receiver operating characteristic (ROC) curve. Such a curve 
is shown in Figxire 13. Any observer whose response plot falls along 
the diagonal of the figure is responding randomly. The responses of 
the observer who responded "signal" too frequently would be represented 
by the y in Figure 13. This observer never misses a signal, but this 
success is at the cost of a high miss rate. The reverse holds for an ob- 
server whose responses fail at the point marked x in Figure 13. 
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Quite obviously, if the magnitude of the signal is increased and 
the noise distribution remains constant, the "hit'' rate will increase, 
even if the criterion values are kept constant. The new curve will fall 
somewhere above the solid curve of Figure 13 and is indicated by the 
dashed curve. The observer's ability to react to the increased signal 
is an index of his sensitivity and is called, in theory of signal detecta- 
bility, his d' value. Quite obviously, various observers will possess 
different d' values. 
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While the majority of signal detection experiments have used 
visual, auditory, or oth(jr sensory stimuli, tho thinking behind sif^nal 
(lcl.(?(:tion th(!Oty, lik(^ threshold comu^pt tliiiikifif^, h is r(»c(*nlly Ixmmi 
applii^d outsido of th(^ r(»;ilin oL' S(uisoi-y psycthophysics, Somi<» oT IJi(»S(» 
typos of application have recently been summarized by Petoi^soa and 
Beach (1967). Peterson and Beach cite examples for the signal detec- 
tion theoretic approach for such areas as perceptual defense, recogni- 
tion, memory, and judgment of the sources of short phrases from a 
magazine. According to these authors, . .these experiments show 
that it is possible to interpret a wide range of psychological phenomena 
within the framework of statistical decision theory. " 

Specifically, Applied Psychological Services (Siegel, Fischl, & 
Pfeiffer, 1968) has demonstrated the utility of d* and related variables 
as criteria for assessing performance and training effectiveness, as 
well as for predicting academic success in both a Navy and a college 
context. 

Using responses to a true-false test situation (ill\js trated in 
Figure 14), these investigators demonstrated that it is possible to de- 
velop ROC curves which reflect differences in training effectiveness. 
Figure 15 presents the result for college students, and Figure 16 pre- 
sents the results for military personnel. 

In the preparation of Figures 15 and 16, the likelihood ratio (LX) 
was computed under the assumption of a normal probability density func- 
tion for signal and noise curves and represents the ratio of the "hit" to 
"miss*' probabilities at the point above which the obseirver placed his 
response criterion. The logic for this procedure has been discussed 
elsewhere by Swets, Tanner, and Birdsall (1964), 

Using the "hit" and "miss" values, the detection sensitivity (d') 
was computed directly from the tables prepared by Patricia Elliott for 
the "yes-no" task, presented as Appendix 1 by Swets, Tanner, & Birdsall 
(1964). The higher the d' value, the greater is the distance between the 
means of the signal and the noise distributions and the less the overlap in 
distributions. Thus, high d^ values represent a sensitive observer, one 
capable of distinguishing, with a low error rate, between signal and nois(j, 
i. e. , one who has experienced effective training. Moreover, it was dam" 
onstrated that the differences noted among the d^ scores at various lov(US 
of training cannot be attributed to differences among the subgroups in basic 
aptitude. Thus, the signal detection tneory seems to possess considerabl(,» 
potential for evaluating training success, 
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Given Answer 


Response of S 




True (A) 


False (B) 






.True (Signal) 


f(S' A) 


f(S- B) 

Q) 


False (Noise) 


f(N-A) 


f(N- B) 


Total (f) 


True 


False 



Total (f) 



f(True) 
f( False) 



Definitions: 



P(AlS) 



f (S- A) 

f(True) 



Hit rate 



False alarm rate 



Figure 14. "True-false^' response classification matrix. 
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P(A|S) 




10 20 30 40 50 60 70 80 90 



p(a|n) 



Figure 16. ROC curves for electronics maintenance personnel 
at three levels of training experience. 
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In the present investigation, a multiple choice eoniidencc test 
was administered to the electronics students. This test was scored 
in both the conventional manner and in accordance with theory of sig- 
nal detection principles. 

Administrative Specialist Tests 

The advanced measurement techniques developed for measuring 
the administrative factors were: (a) partial knowledge, (b) technical 
words, (c) Thurstone scaled typing samples, (d) pictorial absurdities, 
(e) signal detection, (f) need achievement, and (g) psychophysiological 
arousal. The development and use of each of these testing techniques 
are described in detail in paragraphs which follow. All of these tests, 
except need achievement and psychophysiological arousal were oriented 
to test the document preparation factor (factor 1). 

Partial Knowledge Testing 

Traditionally, in scoring a four-choice multiple choice question, 
a student is given credit for the correct answer and no credit for a 
choice of any incorrect answer or distractor. Partial knowledge exists 
when the student can identify one or more of the distractors. With the 
partial knowledge scoring technique in a multiple choice format, one 
point is given for each distractor identified and three points subtracted 
if the correct answer is identified as a distractor. Scores on each four- 
choice item can range from plus three to minus three. Partial knowledge 
testing, as in confidence testing, yields increased item and test variance 
and penalizes for random guessing. Moreover, as compared with the usu- 
al test scoring procedures, it provides a greater source of feedback in- 
formation in regard to areas of student misunderstanding of subject mat- 
ter content. 

The 25 items used in the partial knowledge test sampled the range 
of knowledge required of students in Block II of the course for adminis- 
trative specialists. Twenty minutes were allotted for this test. 

As with confidence testing, students sometimes experience diffi- 
culty learning the idea of selecting the incorrect distractors, rather 
than choosing the correct answer as in the multiple choice format. The 
burden of explanation, then, is placed on the test directions and the ad- 
ministrator. Figure 17 presents the examinee directions employed for 
the administrative specialist partial knowledge test. Sample items taken 
from the partial knowledge test are shown in Figure 18, 
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DIRECTIONS 



This is a test of your knowledge of document preparation. Your 
task is to identify as many of the incorrect answers from among the multi- 
ple choice alternatives as you can. If you identify all three incorrect al- 
ternatives, you will get a higher score than if you only identify two of the 
incorrect alternatives. Identifying two incorrect alternatives gives you 
more points than if you identify only one incorrect alternative. Accordingly, 
for each incorrect alternative you identify, you will be awarded one point. 
Conversely, if you identify the correct alternative as being incorrect, then, 
three points will be subtracted from your score. For example, if you cor- 
rectly identify two of the wrong answers in a four choice multiple choice 
question you receive two points. If, on the other hand, you correctly iden- 
tify two of the wrong answers and incorrectly identify the correct answer, 
then you receive a score of -1 as shown below: 

1 -h 1 - 3 = -1 

If you identify all three wrong answers your score is +3 as shown 
below: 1 + 1 + 1 = 3 



Correct identification of all three wrong answers produces the high 
est possible score on a question. In order to identify the wrong answers, 
simply place a checkmark next to each answer you feel is incorrect. 



Ex ample 

1, Identify the incorrectly spelled word (check the wrong answers). 

a. typwriter 

b. typewriter 

c. ^/ tipewriter 

d. / tipwriter 

In the above example, the student was quite certain that choices £ 
and d were incorrect; therefore he obtained two points for that question. If 
he ha'd chosen choice b in addition to choices £ and d^, his score would have 
been -1, since he incorrectly identified the correct answer as being incor- 
rect. Your best strategy, then, is to check only those answers you are 
certain are incorrect. 



You will have 20 minutes to complete this test. Remember, only 
select the incorrect alternatives, or the alternatives that are most incor- 
rect. The examiner will tell you when to begin. 



Figure 17. Partial knowledge test directions. 
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13, How many net words per minute 
did you type if you typed 183 
strokes, in five minutes, with 
five errors? (check the incorrect 
alternatives) 



16, On air force letters there is a: 
(check the incorrect alternatives) 



a. 
b. 
c. 
d. 



5/5 
" 7/5 
■ -3/5 
"13/5 



a. 
b. 
c. 
d. 



one inch left margin 
one -half inch left margin 
inch and a half left margin 
two inch left margin 



Figure 18. Sample items from the partial knowledge test. 

Technical Words 

Vocabulary in a content area has been a traditional reflector of 
sophistication in the area. Technical words tests generally correlate 
most highly with other measures of achievement. Just as a man^s gen- 
eral vocabulary reflects his general intelligence, his technical vocabu- 
lary reflects his level of accomplishment in a technical area. 

The 32 items in the technical words test sampled the total range 
of knowledge required of students in Block II of the administrative speci- 
alist course. 

In the technical words test, the examinee was presented with a 
word, phrase, or symbol followed by four multiple choice alternatives. 
His task was to identify the multiple choice alternative which is synon- 
ymous with or gives a major use for the word, phrase, or symbol. Fif- 
teen minutes were allotted for completion. Figure 19 presents some 
sample technical words items. 



8. Shift key (check one) 



5. Ampersand (check one) 

a. + 

b. & 

c. # 

d. % 



b. 
c. 
d. 



is used to change lines 
is used to space 
is used to capitalize 
is used to indent 



Figure 19. Sample technical words items. 
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Thur stone Sca le of Typing Ability 



The Thur stone scaling concept, which produces an equal appear- 
ing interval scale along a continuum, has been extensively employed in 
investigations into attitudes but has been seldomly applied for student 
measurement purposes. Yet, the production and use of such a scale 
along a subjective difficulty continuum possesses attractive possibil- 
ities as applied to typing achievement measurement. Such a scale 
would be rooted to the difficulty of the items as perceived by the in- 
structors. Accordingly, it can be argued that a student who achieves 
a given score on a test at level 4 along a Thurstone scaled set of typing 
materials can be considered^ in t!ie view of the instructors, to be twice 
as proficient as the student who receives a comparable score at level 2. 
Moreover, the scaling properties make the conception attractive from 
the point of view of test economy and student placement. From the test 
economy point of view, the student can be moved along the scale in equal 
diffic\ilty increments to find his maximum level in minimum time. From 
the student placement point of view, it affords a direct approach to homo- 
geneous grouping. 

To develop a set of Thurstone scaled typing items, a set of 18 
typing samples of varying difficulty were collected from several sources 
at Keesler /\FB. These typing samples were then submitted to 12 typing 
instructors, at the Administrative Specialist School, who were asked to 
judge the difficulty of each sample along an 11-point scale. The formal 
directions are shown in Figure 20. 

The median difficulty value of each typing sample on the 11-point 
scale was then determined, along with the iterris interquartile range. 

In the Thurstone scaling approach, the median is considered to 
be the scale value of that stimulus. The interquartile range is consider- 
ed to be a measure of uncertainty. Agreement among the judges on the 
scale value produces a low interquartile range value, whereas disagree- 
ment produces a large value. Figure 21 shows the scale and interquartile 
range values of each typing sample. 
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Name 



Today^s Date 



DIFFICULTY DETERMINATION 



You have been presented with a series of typing samples. These 
typing samples vary in difficulty from very easy to very difficult. Your 
task is to assign a difficulty level to each typing sample, Down the left 
hand side of the page is a numerical listing of each typing sample. Across 
from each typing sample number is an eleven point scale. You are to 
place a checkmark (/ ) at that point on the scale that represents the diffi- 
culty level of the typing sample. Try to vary your checkmarks so that 
some appear in all eleven columns. Do not hesitate to use extreme re- 
sponses numbered 1 and 11, if you feel any typing sample deserves one 
of them. 



EXAMPLE 



Very Moderately Moderately Very 

Easy Easy Difficult Difficult 

12 345 6 789 10 11 



Sample 1 
Sample 2 



The first check indicates that the judge assigned a Moderately Easy 
difficulty level to typing Sample 1, The second check indicates that the 
judge thought typing Sample 2 was Very Difficult , 



Figure 20, Direction sheet for Thurstone scaling of typing samples. 
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Sain|)l(»s I, 10, (J, iiiid U v^rvc st^JccLt^d lor iiulusion into llio ad- 
ministrative specialist test battery. These typing samples have scale 
values of 3.5, 5.5, 7.5, and 10.0 with interquartile range values of 
2.0, 2,5, 2.5, and 1.0 respectively. We note that (see Figure 21) the 
upper bound of the interquartile range value for any selected sample 
does not overlap with the lower bound of the next selected sample. 

Typing sample number 6 which had a difficulty value of 7,5 is 
shown in Figure 22, Ninety seconds were allowed for completion of 
this typing sample. The typing samples were administered to the stu- 
dents in order of difficulty from easiest to most difficult. Rather 
stringent time limits were set to ensure that no examinee would com- 
plete any typing sample before the allotted time limit. Otherwise, faster 
students would be penalized. 



•••••••••••••••••••• " ' Jc un t 

Cn th'^ :ncrnin : of 2'f Anril -.he riermans counter attijcked " ith a 69 
force estimaied at Ocit!:alions -'f infantry sup':orteri by artillery. 

The att'jck la 53 ted fro^i 0-00 to 1?00, during 'vhich time the enemy made 20fi 

assaults from rr'.ctically every direction. In this action the ?d 27'* 

'•^attalion lost 21 dead and 11 wounded. The gallant and intre^'id 3^0 

conduct of this entire battalion afforded a :^rea t tactical advanta -^e ^09 

in seizin-^ terrain and in zhe defeat of German arms in North Africa. ^^7?- 



}?....:.... f?... ^9... :...^?o... :...!?•... 

Figure 22. Thurstone scaled student typing sample. 
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Pictorial Absurdities 



In the administrative absurdities test, the student was required 
to examine 10 pictures or documents. Each contained some incorrect 
action or facet. The examinee^s task was to state what is wrong in 
each picture. Fifteen minutes were allotted for this test. The advan- 
tages of this type of test were discussed under the prior description of 
the electronic principles tests. A sample item from this test is shown 
in Figure 23, 

Signal Detection 

With regard to the Administrative Specialist tests, the partial 
knowledge test was scored using signal detection principles. The the- 
ory of signal detection was discussed, at length, in the descriptions of 
the electronic principles advanced measurement techniques. 
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Figure 23. 



Sample administrative absurdities item. 
65 



Auxiliary Assessment Methods 



The remaining two measures used in the current investigation 
were not tests, per se, but are novel measures which might predict or 
moderate future performance or which might provide training feedback 
information beyond that usually available. These two additional meas- 
ures involve the concepts of need achievement and psychophysiological 
arousal. 

Need Achievement 

An airman's motivation, whether in the school situation or in 
the field, can play an important a role as a success determinant. Ac- 
cordingly, it seemed important to explore whether or not a measure 
of motivation could make a significant contribution to the goals of the 
present study. The measure of motivation employed was the desire or 
drive of the student to attain a prestigious position in general and in the 
Air Force. 

Atkinson and O'Connor (1963) correlated the prestige value of 
occupations with probability of success estimates given by high school 
students. The resultant correlation was found to be , 90, In a similar 
study, Strodtbeck, McDonald, and Rosen (1957) found that Jewish boys 
would be more satisfied with high status occupations (e.g., doctor, 
executive) and less satisfied with low status occupations (e,g, , night 
watchman) than Italian boys. In other words, a steeper slope of satis- 
faction was found for Jewish boys than for Italian boys. These authors 
also found a steeper slope of satisfaction for middle class boys than for 
lower class boys. Atkinson (1966) feels that the Strodtbeck, McDonald, 
and Rosen study is V.. particularly exciting, because independent studies 
have shown n Achievement to be stronger among Jews than Italians and 
stronger in the middle class than in the working class (or upper class) 
[p. 164]. 

A short "need achievement" questionnaire was developed and 
administered to both the electronics and the administrative samples. 
The data yielded by this questionnaire allow derivation of insight into 
the relationships between student motivation to succeed and tost scores. 
The questionnaire used employed the methods and techniques described 
by Morgan (1966) which were based upon the Strodtbeck, McDonald, and 
Rosen (1957) study. 
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In the questionnaire, the students were required to compare 
the desirability of various civilian and Air Force occupations on a 
six point scale. Then, the regression or slope of the individual on 
the group is calculated. The resultant slope index is the index of 
achievement motivation. Accordingly, the greater the covariance be- 
tween the individiial's occupational hierarchy and that of the group, the 
higher his achievement motivation index. The civilian occupations 
questionnaire is shown in Figure 24, 

OCCUPATIONS QUESTIONNAIRE 



Name (print) Date 

Directions 

We are interested in how people compare occupations. How do you 
think most people would feel if a friend or relative of theirs chose certain 
types of work? Decide whether most people think each of tiie listed occu- 
pations is: very desirable (VD), desirable (D), slightly desirable (SD), 
slightly undesirable (SU), undesirable (U), or very undesirable (VU), 

Civilian Occupations 
(Circle one for each occupation) 



a. 


Night Watchman 


VD 


D 


SD 


SU 


u 


vu 


b. 


Carpenter 


VD 


D 


SD 


SU 


u 


vu 


c. 


Mail Carrier 


VD 


D 


SD 


SU 


u 


vu 


d. 


Bus Driver 


VD 


D 


SD 


SU 


u 


vu 


e. 


Bookkeeper 


VD 


D 


SD 


SU 


u 


vu 


f. 


Druggest 


VD 


D 


SD 


SU 


u 


vu 


g. 


Teacher 


VD 


D 


SD 


SU 


u 


vu 


h. 


Doctor 


VD 


D 


SD 


SU 


u 


vu 


i. 


Assemblyline Worker 


VD 


D 


SD 


SU 


u 


vu 


j. 


Janitor 


VD 


D 


SD 


SU 


u 


vu 


k. 


Office Marager 


VD 


D 


SD 


SU 


u 


vu 



Figure 24, Civilian occupations need achievement 
questionnaire. 
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Psychophysiological Arousal 

The psychophysiological measure used in this investigation \vas 
the proportion of active sweat glands on a four millimeter square ai'oa 
of the central whorl on the finger tip of the third finger of the less used 
hand. The technique was developed by Sutarman and Thompson (1951) 
as a substitute for the galvanic skin response (GSR) and has since been 
improved by Johnson and Dobbs (1967). Only one investigator (Martens, 
1969) has successfully used it as a test of social facilitation theory. 
Another investigator (Bergman, 1971) used it to assess the physiologi- 
cal response to crowding and test failure. 

To employ the technique, a solution is applied to the finger tip of 
the subject with a cork stopper. After a 15 or 20 second interval, the 
solution dries and is removed with Scotch Magic Transparent Tape and 
mounted on millimeter ruled graph paper. When the print is placed 
against a lighted surface and subjected to 10-20 power magnification, 
the number of active and inactive sweat glands on a demarcated area 
of the sweat print can be counted. The proportion of active sweat glands 
was used as the criteria rather than the number of active sweat glands 
since the total number of sweat glands on a finger tip can vary from in- 
dividual to individual. The chemicals used to make the sweat sensitive 
compound were as follows: 

1. five grams of polyvinyl formal which withdraws from 
moisture 

2. ten milliliters of butyl phthalate which gives the print 
toughness 

3. twenty grams of a semi-colloidal dispersion of graphite 
in trichloethylene which provides optical contrast 

4. one-hundred milliliters of ethylene dichloride which is 
a solvent 

The inter scorer reliability coefficients for the number of active sweat 
glands (Johnson & Dobbs, 1967) were reported to range from .87 to .98, 
and the reliability of one scorer who made counts several days apart was 
.99. 
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Johnson and Dobbs (1967) report increases in palmar sweating 
before and after e administration of mathematics tests to students 
and a decrease in sweating during the tests. Harrison and MacKinnon 
(1962, 1966) found that palmar sweating decreased while an individual 
suspended an unsupported limb in the air, and that the anhydrous re- 
sponse was mediated by the adrenal medulla. 

Since many persons exhibit emotional arousal in testing situa- 
tions, it was thought elevant to include a measure of psychophysiologi- 
cal arousal in this investigation. As with need achievement, arousal 
may account for some portion of the test score variance. On difficult 
or un^" .miliar tasks, the performance of highly aroused subjects shc uld 
be disrupted, but moderate arousal should be facilitating. Similarly , 
no arousal at all should exert a aegative effect on performance. Appli- 
cation of the solution is quick and easy, and the technique is much 
less obtrusive than other arousal measures (e,g, , GSR) [Bergman, 
1971], 



Qu estionnaires and Interviews 



Following completion of the testing, each student was required 
to complete a structured questionnaire regarding the desirability or un- 
desirability of each of the advanced measurement techniques employed. 
These questionnaires are reproduced in the appendix to this report. 

Additionally, instructors from the electronics school and instruc- 
tors from the administrative school were given individual briefings on the 
development, rationale, and use of each advanced measurement terhnique. 
At the completion of the briefings, 10 instructors from each school were 
individually interviewed regarding the probable effectiveness of the 
new techniques as compared with the old, the utility of the novel tech- 
niques for assessing student attainment of training objectives, admin- 
istrative and logistic considerations, and possible effects of the new 
techniques on the role of the instructor and the training evwduation 
specialist. 
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Data Treatments and Criteria Employed in 
Judging the Value of the New Instruments 



Correlational Analysis 

To test the merit of the novel test instruments, uach of the ad- 
vanced measurement techniques and the auxiliary measures was con- 
sidered as a predictor within a concurrent validational effort. The 
criteria in this validation effort were final average and examination 
grades in the respective courses. The advanced measurement tech- 
niques were anticipated to account for a significant proportion of the 
final average and examination grade variance. As an ultimate test of 
the merit of the novel measurement techniques, predictive validity 
would have to be established. However, we note that such predictive 
validity has not been established for the usual course examinations. 
Moreover, it seems that because the novel instruments tap a greater 
range of cognitive function, on the surface they should possess greater 
predictive validity than the currently employed measures. It seems 
obvious that posttraining field performance involves a greater range 
of intellective activity than is measured by the usual multiple choice 
tests. 



Reliability 

The second test of the merit of the novel measures was concerned 
with test reliability. Where appropriate, reliability estimates based on 
the Kuder-Richardson (formula 20) (split half methods were employed in 
those instances in which the Kuder-Richardson formula could not be util- 
ized) and Spearman-Brown methods were applied to each advanced meas- 
urement technique, 

Cost/Benefit Analysis 

The third test of the value of the present advanced measurement 
instruments involved a cost benefit analysis. The cost/ benefit analysis 
relied on the derivation of a "figure of merit" on the basis of the follow- 
ing formula: 
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Cost/Benefit = |^ j^i J x or 

where: DC = Direct costs 

AC = Administrative costs 

SC = Support costs 

N = Number of students evaluated 
2 

r = Test-criterion variance 

S = Number of course improvement suggestions elicited 
This formulation was applied to each test type separately by course. 
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Ill RESULTS 



The first sot of results to be presented are those dealing with 
the electronic measures. These results will be followed by a pi'csen- 
tation of the administrative specialist results. Each set of results is 
composed of five parts, including: (a) correlational analysis (concur- 
rent validity), (b) reliability, (c) student questionnaire results, and 
(d) cost/ benefit analysis. 

# 

Electronic Principles Measures 



Electronic Principles Correlational Analysis 

Each of the advanced measurement techniques was used as a 
predictor of school performance in a concurrent validity study. The 
auxiliary measures (stress measure and need achievement measure) 
and the scores on the usual school Block IV multiple choice tests were 
also included as predictors in this phase of the analysis. The criterion 
of school performance was the average of the final grades for electronic 
principles in Blocks I-VII excluding Block IV. (Only the first seven blocks 
of instruction were considered, inasmuch as these were the only blocks 
that all the students had in common. ) The Block IV exam was not in- 
cluded as part of the criterion average since it was included as a pre- 
dictor. This validational paradigm is analogous to item analytic tech- 
niques in which each item is correlated with total test score after the 
item^s contribution to that test score has been removed. 

As a second step in the correlational analysis, the method of 
stepwise regression (Dixon, 1965) was employed. Stepwise regression 
computes a series of . . multiple linear regression equations. ... At 
each step one variable is added to the regression equation. The vari- 
able added is the one which makes the greatest reduction in the error 
sum of squares. Equivalently, it is the variable which has highest 
partial correlation with the dependent variable partialled on the vari- 
ables which have already been added; and equivalently it is the variabh* 
which, if it were added, would have the highest F value" [p. 233]. 
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Prior to completion of the stepwise regression analysis, sev- 
eral of the advanced measurement technique scores were either multi- 
plied by a constant and/ or had a constant added to them. This was per- 
formed in order to eliminate decimals and negative numbers. These 
measurement techniques with their additive and/ or multiplicative con- 
stants are shown in Table 26, 



Table 26 



Additive and/ or Multiplicative Constants for the Electronic 
Principles Advanced Measurement Techniques 




Add 1 1 1 ve 


Multiplicative 


Te ch n 1 q u e 


Con Slant 


Con scant 


Figural Systems 


101. 00 




Schematic Reading 




10.00 


Confidence Score 


134.00 


10. 00 


Confidence Test (d») 


1.55 


10. 00 


Analogies 


8.00 


10. 00 


Achievement Motivation 




10. 00 



The means and standard deviations of the electronic advanced 
measurement techniques, the auxiliary measures, the Air Force apti- 
tude variables, the Block IV score, and the criterion variables are shown 
in Table 27. The intercorrelation matrix for these same variables is 
presented in Table 28. 

From the point of view of the concurrent validity of the advanced 
measurement techniques, we note that, with the exception of the absurd- 
ity tests (where no validity was shown), moderate to rather high coeffioi- 
ents are indicated in Table 28. The conventional score, confidence 
score, and d* m(?asures exhibited particularly high validity coeCficic^nts- - 
. 64, .58, and .64 rospoctivoly. The remaining advanced moasuronionl 
techniques exhibited validity coefficients which ranged from . 28 (anal- 
ogies) to . 42 sequential test level. These are interpreted as moderate 
and acceptable. The auxiliary measures, need achievement and psycho- 
physiological arousal, exhibited no predictive validity whatsoever. 
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Table 27 



Mean and Standard Deviation of the Advanced Measurement Tech- 
niques, the Auxiliary Measures, * le Air Force Aptitude Variables, 
the Block IV Score, and the Criterion Variables for 41 Students 

Enrolled in the Electronics Principles Course 



Variable 


Mean 


Standard Devi 


Sequential Test Level 


2.829 


1. 093 


Confidence Test Score 


159.073 


65. 999 


Signal Detection Test (d') Score 


196.585 


67. 697 


Conventional Test Score 


15.439 


5. 201 


Pictorial Absurdities 


3. 268 


2. 398 


Pictorial Schematic Reading 


85.341 


22. 742 


Cognition of Figural Systems 


48. 244 


17. 121 


Analogies 


43.049 


19. 497 


rieed Achievement (Civilian) 


144.220 


84.641 


Need Achievement (Air Force) 


132.561 


28. 584 


Psychophysiological Arousal 


73.366 


26. 261 


General Aptitude 


78, 780 




Mechanical Aptitude 


77.073 


12.989 


Administrative Aptitude 


71, 098 


17. 906 


Electronic Aptitude 


86, 341 


5, 365 


Block IV Exam Score * 


77.512 


12. 789 


Criterion 


79.463 


7. 349 



* Solid State Power Supplies and Amplifiers 
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In order to bring organizational clarity to the data of Table 28, 
the table was cluster analyzed. One of the advantages of a cluster 
analysis, in this context, is that it can demonstrate v/hy some of the 
variables which are highly correlated with the criteiion are either not 
present in the multiple regression equation or contribute only a mini- 
mal amount of variance to multiple prediction of tl^e criterion score. 
The results of the cluster analysis are shown in Table 29. 

The largest most highly interrelated cluster consists of four 
methods bailed on items from the Block IV tests. One would expect, 
+hen, a great deal of variancf o/erlap between these tests. The 
fourth measure (sequential test) in this cluster consists of different 
Block IV items from the same tests. Surprisingly, the usual Block 
IV exam score failed to fall into this cluster. 

The next largest cluster consists of tlie electronic, general, 
and mechanical aptitude tests from the Air Force aptitude battery. 

The third cluster consists of the psychophysiological arousal 
Txieasure and the civilian need achievement inventory. A low score 
on the psychophysiological measure indicates high arousal while the 
need achievement score was polarized in the opposite direction. 
Those two variables are inversely related to each other at a value of 
-,44. This finding suggests, as might have been anticipated, that 
airmen who exhibit considerable physiological arousal are also high 
in achievement motivation. Conversely, airmen who exhibit little 
physiological arousal during test taking are low in achievement moti- 
vation. These two tests exhibited insignificant, but consistent, rela- 
tionships with the mechanical aptitude, the electronics aptitude, and 
the Block IV test scores. The need achievement index was corrt^lat^d 
-, 24, 27, and 22 with each of these three tests respectively, while 
psychophysiological arousal correlates ,20, . 13, and . 24 with the 
three tests. The above results seem to be consistent with current 
motivational theory. Apparently, a certain degree of arousal is nf-c- 
essary for motivated behavior. 

The final cluster consists of the figural systems and analogies 
tests* Both of these tests require a high degree of abstract thinking by 
the examinee. What is unusual about both of these tests is that they 
contain items which are almost devoid of verbal content. Although these 
tests are nonverbal, they still correlated with the criterion . 38 and . 28, 
respectively. 



77 



CO 

o 



? £ 2 



^ 2 o o <J> IS 







go. 


m 




o 








ce 








<p 




u. 




o 


10 


w 




(0 












< 






a: 


< 


UJ 






q: 


:e 


tii 






Z 


(/) 


UJ 


3 


-J 


-J 


0) 


U 


< 
















Analogies 

Cognition of Figural Systeis 

.Arftinistrative Apcitud? 
Need Achievenent (Air Force) 

Electronic Aptitude 
General Ap ti tude 
Mechanical Aptitude 

Psychophysiological Arousal 
Need Achtevesent ("civilian) 

Confidence Score (d) 
Confidence Score 
Conventional Test Score 
Sequential Test Level 

Schematic Reading 

Block "V Exa«ination Score 
Absu rd i ties 



(X> ^ to ro 

a^ CO ^ ^ 



CV! 0) to 

2 o 
^ ""^^ ^ 



0> 0> CM 2! 



ERIC 



7H 



The remaining five tests (absurdities. Block IV examination 
score, schematic reading. Air Force need achievement and adminis- 
trative aptitude) did not seem to fall into any meaningful cluster. 

The results from all 12 steps of the stepwise regression pro- 
cedure (Dixon, 1965) are shown in Appendix A of this report. In this 
section we present the one, four, and eight factor equations so derived. 
Each of these regression equations is shown in Table 30, 

Table 30 

Multiple Regression Equations for Steps One, Four, and Eight of 



Stepwise Regression Procedure (Electronic Principles Course^^ 



Step 




Regression Equation 


Multiple R 


one 


y' 


= 65. 586 + . 899X5 


.636 


four 


y' 


= 34. 843 + . 183Xg + . 042x^ + . 287x^^ + . 1 12x^g 


. 713 


eight 


y' 


= 40. 430 + 1. 041X1 + . 066x + . 044x - . 018x 
+ .176x^3-. .118x^3. 


. 762 



sequential test level 

schematic reading/ component identification 
conventional test score 
confidence score (d') 
civilian need achievement 
electronics aptitude 
Block IV exam score 

Table 30 further supports the prior contention favoring the valid- 
ity of the novel measures. Note that the one variable equation yielded 
an r of . 64, The Block IV examination did not become included in the 
equation until step 4, 



* -1 

^3 

X 



5 
X7 

Xg 
^15 

Xl6 
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Variable 5, conventional test score, was eliminated as a pre- 
dictor in the latter multiple regression equations. This variable was 
replaced by variable 1, sequential test level. The reader will recall 
that variable 1 clustered with variable 5 in the cluster analysis. How- 
ever, the intercorrelation matrix in Table 29 shows that variable 1 i-5 
less related to the other variables in the matrix than is variable 5. The 
variance that variable 1 has in common with the criterion is almost 
totally unique, while the variance that variable 5 has in common with 
the criterion can be accounted for by other variables including vari- 
able 1. 

Variance analytic results for the statistical significance of the 
linear prediction for steps one, four, and eight are shown in Table 31. 
All the F tests are statistically significant. 



Table 31 

Analysis of Variance Summary for the Significance of Linear Prediction 
for Steps One, Four^ and Eight of the Stepwise Regression Procedure 



Analysis of Variance 
df Sum of Squares Mean Square F 



1 


Regression 


1 


874. 288 


874. 288 


26.516* 




Residual 


39 


1285. 907 


32. 972 




4 


Regression 


4 


1098. 570 


274. 642 


9. 313* 




Residual 


36 


1061. 625 


29. 490 




8 


Regression 


6 


1254. 142 


209.024 


7. 844* 




Residual 


34 


906. 053 


26.649 





*p < .01 
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Sigrial Detection 



In the present investigation, the signal detection method of scor- 
ing the confidence test did not appear to yield a validity coefficient that 
is appreciably different from the confidence test score, or the conven- 
tionally scored test. 

As will be recalled, the theory of signal detection provides a 
method for controlling and measuring the criterion an observer uses 
in making decisions about signal existence and it provides a measure 
of the observer's detection sensitivity (d') that is independent of his de- 
cision criterion, "^"he sensitive observer is one who differentiates with 
few errors between Ignal and noise. 

(23 

A receiver operator characteristic (ROC) curve depicts d* as a 
function of hit rate (HR) and false alarm rate (FAR). In the present con- 
text, HR is defined as the proportion of time the subject correctly iden- 
tifies signal when presented with signal plus noise at a confidence level 
of . 68 or higher. (The results of a significance test determined that in 
a four choice multiple choice item confidence values up to .68 are not 
significantly different from chance (. 25) confidence estimates. Confi- 
dence estimates above . 68 are considered significant beyond the chance 
level. ) FAR is defined as the proportion of time the subject incorrect- 
ly identifies noise as signal when presented with noise only at a confi- 
dence level of . 68 or higher. When drawing a ROC curve, the HR is 
depicted along the ordinate, while the FAR is depicted along^ the abcis- 
sa. The ROC curve shown in Figure 25 presents the average d* for sub- 
jects scoring the top, middle, and bottom thirds of the criterion group. 
The reader will note the large degree of differentiation between the three 
ROC curves indicating an appreciable degree of discrimination. It is 
possible that test score improvement could be a+tained by students who 
score in bottom third if they adjusted their accepiw..ice criterion upward. 

Reliability Analysis 

The Kuder -Richardson internal consistency reliability formula 
(formula 20) was applied in all appropriate cases to the electronic prin- 
ciples tests. In some instances (e.g. , confidence test scores, cognition 
of figural systems test) split half methods were utilized because the data 
could not be adopted to the Kuder-Richardson algorithm. Reliability co- 
efficients were approximated for tests of 25 items (except for the confi- 
dence test which already contained 30 items) using the Spearman-Brown 
formulation. A test length of 25 items was used, inasmuch as this 
length seemed to be acceptable in view of testing constraints in the mil- 
itary School situation. 
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FALSE ALARM RATE 

Figure 25. Det c'.or sensitivity (d') for stude.its scoring in 
the bottom, middle, and top thirds of the criterion. 
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Table 32 presents the corrected reliability coefficients for the 
tests taken by the electronic principles students. When co.nparing 
tests developed on the basis of items constructed by Applied Psycho- 
logical Services with tests using electronics school items, it seems 
as though the reliabilities of the latter are only marginal, while the 
reliabilities of the former range from marginal to high. These re- 
sults suggest a need for school examination of the internal consistency 
reliability of their end of block tests. Reliability coefficients of . 6 - . 7 
are certainly marginal. 



Reliability Coefficients for Naval Electronics Principles Measures 



Table 32 



Test 



n 



Coefficient 



Sequential Test i 
Sequential Test II 
Sequential Test III 
Sequential Test IV 
Sequential Test V 
Confidence Test Score 
Conventional Test Score 
Pictorial Absurdities 
Pictorial Schematic Reading 
Cognition of Figural Systems 
Analogies 

Psychophysiological Arousal'*' 



3 
13 
44 
8 
6 
44 
44 
44 
42 
44 
44 
30 



.82 
.65 
.68 
.79 
.60 
.64 
.85 
.67 
.84 
. 88 
.99 



* Interscorer reliability 
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Questionnaire Analysis 



Following completion of the testing, each student who had parti- 
cipated in the electronics testing completed a structured questionnaire. 
The questionnaire (presented in Appendix E) sought to determine student 
reaction to each of the advanced measurement techniques. The 16 items 
included in the questionnaire concerned the relevance, fairness, prefer- 
ence for, complexity, and interest of the various advanced measure- 
ment techniques when compared with the usual multiple choice school 
tests. The student was requested to indicate his agreement or disagree- 
ment with each item on a six point scale. Mean responses, across all 
students, were then obtained. Table 33 presents the mean value, topic, 
and question number for the questionnaire data. In developing Table 33, 
all scales that were presented to the students in the reverse direction 
(approximately half of the scales were originally reversed in order to 
avoid the effects of response set and acquiescence) were again reversed 
to yield consistency within the table and ease of comparison. In addi- 
tion, the items are regrouped according to test or test type for ease of 
comparison. Mean values between 3. 00 and 4. 00 on the scale indicate 
neutrality of opinion regarding the advanced measurement techniques. 
Values greater than 4. 00 indicate that the students responded positive- 
ly to the advanced measurement technique as compared with the usual 
school tests, while values less than 3.00 indicate that the students re- 
sponded unfavorably to the advanced measurement techniques* Ques- 
tions concerned with the sequential tests and other conventionally 
scored test^ were not included since they represent the usual multiple 
choice school tests. 

Generally, Table 33 suggests that the more novel approaches 
(e.g. , cognition of figural systems, absurdities) were less preferred 
than the usual school tests. Conversely, novel approaches which repre- 
sented refinements of traditional testing procedures (e. g. , confidence 
testing, schematic reading) were considered fairer and more relevant. 
There are several reasons why the students may have tended not to pre- 
fer the more novel testing techniques. First, many individuals will 
react with fear and dislike when presented with anything new or unusu- 
al. They will often feel incompetent and unable to cope with the new 
situation. Given time and experience, though, attitudes often change. 
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Table 33 

Mean Response to Questionnaire Items for Various Tests 



Question 



Number 


Test 


Topic 


Mean 


1 


Confidence Test 


Fairness 


4.49 


6 


Confidence Test 


Preference 


3.68 


5 


Pictorial Absurdities 


Preference 


2. 15 


3 


Pictorial Schematic Reading 


Relevance 


4.07 


8 


Pictorial Schematic Reading 


Preference 


3.49 


10 


Cognition of Figural Systems 


Preference 


1.85 


13 


Cognition of Figural Systems 


Preference 


2.51 


2 


Analogies 


Relevance 


2.68 


7 


Analogies 


Preference 


2.66 


4 


All Advanced Measurement Techniques 


Interest 


3.73 


9 


All Advanced Measurement Techniques 


Difficulty 


3.68 


11 


Advanced Measurement Test Directions 


Difficulty 


5. 17 


12 


All Advanced Measurement Techniques 


Interference 


2.98 


14 


All Advanced Measurement Techniques 


Interest 


3.54 


15 


All Advanced Measurement Techniques 


Complexity 


4.02 


16 


All Advanced Measurement Techniques 


Relevance 


3.98 



Second, the students indicated the experimental testing program to inter 
for with their daily school schedule, and for this reason, they may have 
reacted negatively toward some of the more unique measurement tech- 
niques. Fin£dly, the novel measures were more abstract and difficult, 
and they provoked the individual to use higher order thinking processes 
and concepts. The novel methods did not rely on simple stimulus-re- 
sponse situations as is the case with most multiple choice tests. A 
iest that requires more work on the part of a student is less likely to 
be preferred. 
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Interview Analysis 



Ten instructors from the electronics school were given individu- 
al briefings on the development, rationale, and use of each advanced 
measurement technique. Each instructor was then systematically in- 
terviewed regarding his assessment of the effectiveness of the new tech- 
nique, its utility in regard to assessing student attainment of training ob- 
jections, administrative and logistic considerations, and the possible 
effects of the novel approaches on the role of the instructor and train- 
ing evaluation specialist. Tables 34-39 present the instructors* eval- 
uation of the effectiveness of each of the new advanced measurement 
techniques when compared with the usual multiple choice school tests. 
Effectiveness was rated on a 0-100 scale with a value of 50 represent- 
ing effectiveness equal to that of multiple choice tests. A value greater 
than 50 indicated that the instructor considered the advanced measure- 
ment technique to be more effective than the usual multiple choice elec- 
tronics school test, while a value of less than 50 indicated that the in- 
structor thought the advanced measurement technique was less effective 
than the multiple choice school test. Value judgments between 45 and 
55 indicate a neutral judgment. 

The results presented in Tables 34-39 indicate that the instruc- 
tors, as a group, had a favorable opinion of the sequential test (because 
of its ability to improve the channeling of students), the confidence test 
(because of the increased information it provides), the figural systems 
test (because it tests understanding), and the schematic reading test (be- 
cause a similar format is currently used). The instructors seem to 
have had a neutral opinion of the analogies test and an unfavorable opin- 
ion (for a variety of reasons) of the absurdities test. These two tests 
also have been shown earlier to exhibit low validity coefficients. 



86 



Table 34 



Evaluation of Effectiveness of the Sequential Tests as 
Compared with Multiple Choice Tests by Ten Electronics 

Inst/'uctors at Kc^oslor Air I*'orce Buso 

Instrutlor 

Number Effectiveness Comments 

1 90 A hell of a lot more effective. 

2 75 No comment 

3 60 No comment 

4 83 This will be a very effective method for 

channeling students on the basis of test 
results when we go over to self pacing 
next year. 

5 75 For self pacing it will be better than what 

we have now. 

6 50 Can't see any more or less advantage. 

7 75 You can advance him into something else. 

You*re not stuck with one test. 

8 50 Same in effectiveness. 

9 80 You can find out which students learn the 

material most quickly and you can get 
them onto something else. You identify 
the person who needs help. Now, they 
teach at the lowest level demotivating many 
students* 

10 40 Must stick to lock step, this is most adapt- 

able to self pacing. Because what weVe 
got now where students are given certain 
material on a certain day. 

Mean 67.80 
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Table 35 



Evaluation of Effectiveness of the Confidence Test Conapared 
with Multiple Choice Tests by Ten Electronics Instructors at 

Keesler Air Force Base 

Instructor 

Number Effectiveness Comnaents 

1 75 Ease of grading. It does give better picture of 

how much he knows and removes guess factor. 
More difficult to grade 

2 70 No comment 

3 90 No comment 

4 75 If students had time to learn how to use it, 

5 75 Might help but don^t know if it can completely 

test a man's knowledge. Would not want to 
use it instead of straight objective tests. It's 
better for diagnostic purposes. . 

6 38 I want to know if the student is sure of the 

right answer. If he's only partly sure that's 
not good enough. He's got to be able to perform 
on the job, 

7 75 It might be good with some of the test items, 

but not with all. Sometimes he must know it 
all to satisfy a criterion objective. 

8 90 It's more importaint to be sure of an answer 

than to answer the question right. 

9 30 Less effective. Confusion among students. A 

good student will underrate himself and a poor 
student will overrate himself. 

10 95 Gives you differentiation between students 

who know the material and those guessing. 
Gives credit where credit is due. 



Mean 71.30 
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Table 36 



Evaluation of Effectiveness of the Absurdities Test as Compared 
with Multiple Choice Tests by Ten Electronics Instructors at 
^ Keesler Air Force Base 

Instructor 

Number Effectiveness Comments 

1 35 His job is not to build a circuit, but to work 

with what he has. He'll work on a complete 
circuit and troubleshoot on that circuit. Not 
realistic. 

2 25 Too much memorization and recognizing 

circuitry. 

3 65 No comment 

4 25 We don't instruct at that high a level. We teach 

basic principles of electronics. Good in field. 

5 50 Similar to troubleshooting in the lab. 

6 25 Alone, against standard tests it would be less 

effective. Students called it ''dopey" test, 

7 25 They would have to recall what circuit should 

look like. 

8 25 We don't like to show students something that is 

wrong, because he might retain this. This 
could be an informative test tool for advanced 
training not down on the elementary level, 

9 40 Too time consuming. Con't use other knowledge. 

It's a total recall test. 

10 35 Good in some situations. Measures observation 

rather than information. 

Mean 40.00 
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Table 37 



Evaluation of Effectiveness of the Schematic Reading/ Component 
Identification Test as Compared with Multiple Choice Tests by 
Ten Electronics Instructors at Keesler Air Force Base 

Instructor 

Number Effectiveness Comments 

1 75 Test followed objectives well. Not too different 

from the tests we have now. Questions got 
right to the point. 

2 50 About the same as what we are doing now. 

3 50 Similar to what is used now.' 

4 50 It's almost the usual school tests. 

5 50 Same 

6 55 It's good. Used in part of a system test. Would 

not want to use it alone. It's more effective in 
some places. It gets at concepts and helps to 
test degree of knowledge about the circuit. 

7 50 We do about the same thing today, especially 

when teaching the circuit. 

8 5 0 No comment 

9 70 You're going to have to know what component 

does and current flow. You have to know circuit 
to be able to connect it and for it to work and 
this is what they do on the job anyway. Job 
relevant. 

10 70 Because they have to know it. Troubleshooting. 

IVIean 57.00 
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Table 38 



Evaluation of Effectiveness of the Cognition of Figural Systems Test 
as Compared with Multiple Choice Tests by Ten Electronics Instructors 
at Keesler Air Force Bas e 

Instructor 

Number Effectiveness Comments 

1 30 Objective is circuit identification, not 

building a circuit. 

2 25 It's a difficult test considering we have a 

basic electronics course. We can't teach him 
that much material in the length of time allotted. 

3 70 Good system 

4 75 No comment 

5 25 No com.ment 

6 75 I like it. A good way to test students in cir cuit s. 

It tests man's ability to analyze, logical Lliink- 
ing and knowledge at the same time. It should 
not be used alone, but as part of a test system. 
A good test. 

7 80 They will really have to study a circuit to see 

how it is wired together. It will require the 
students to learn more because in the past the 
teacher told them the relationship between parts. 
Now they will have to do it. 

8 10 We don't teach or have them memorize it. If we 

emphasized practicing a procedure when they 
can know the exact arrangement of thing^s, maybe, 

9 75 You're going to have toknow what component does 

and current flow. You have to know circuit to be 
able to connect it and for it to work and this is what 
they do on the job anyway. Job relevant. 

10 ^ 55 Identifying schematic is important. This makes 

sure they know it. It overemphasizes something 
we don't irtend to emphasize. 



Mean 57.00 
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Table 39 



Evaluation of Effectiveness of the Analogies Test as Compared 
with Multiple Choice Tests by Ten Electronics Instructors at 
Keesler Air Force Base 

Instructor 

Number Effectiveness Comments 

1 65 Makes the student stop and think. Can't guess. Will 

make student sit down and study or he hasn't a chance. 

2 25 Won't test what he needs to learn. It's just comparing 

pictures. "Vlemorization rather than knowing and under 
standing a circuit. 

3 10 Because wouldn't be able to work with it. They don't 

know that much about it. Too much partial visualiza- 
tion required. 

4 70 No comment 

5 40 A little harder. 

6 25 Use of this test would be limited. May be good for 

some things like circuit recognition. As one of a 
gang of tests. O.K. as far as it goes would be less 
effective than straight multiple choice. 

7 50 Correlation between two different things is used 

today in some way. 

8 50 We do this on a very limited scale, presently, but 

we don't have a test full of this testing approach. 

9 50 Too time consuming. 

10 85 Some questions were ambiguous. You're not testing 

reading or wording, only what he knows about 
electroniijs. 

VIoan 47.00 
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Table 40 j/r('S<MiLs Ihr iMsl.rucl.ofs ' csU in.n ions of Hu* |)f<)|);rl)ilil,y 
that administi'alivo oi* loj^isLics problems would arise* in the use* of l.lie 
advanced measurement techniques. The instrueioi s woi'c asked l.o es- 
timate the chances in 100 that these kinds of problems would arise; 
therefore, the higher the mean value sho^vn in the table, the greater^ 
the instructors' estimation that that administrative and logistics prob- 
lems would arise. 



Table 40 



Estimation by 10 Electronics Instructors of the Probability 
that Administrative or Logistics Problems Would Arise in 
the Use of the Advanced Measurement Techniques 



Technique Probability 

Sequential Tests . 46 

Confidence Tests . 40 

Absurdities Test « 43 
Schematic Reading/ Component Identification Test . 29 

Configuration of Figural Systems Test . 62 

Analogies Test . 36 



Table 40 reveals that the schematic reading test estimate is the 
only one with a low mean probability value. All of the remaining tests 
exhibit roughly twice the problem probability of the schematic reading 
test. The schematic reading test is similar in format to an already em- 
ployed test. The particular logistical and administrative problems fore- 
seen by the instructors centered on three areas: (a) need for more per- 
sonnel, (b) grading and scoring problems, and (c) time consumption. It 
seems, however, that these problems must be weighted against any in- 
creased value for the novel measures, if employed. 

A final question asked the instructors about the effects of the 
advanced measurement techniques on themselves and the training eval- 
uation specialist. The most pervasive effect of the advanced measure- 
ment techniques, as indicated by the instructors, was that they would 
necessitate a change in teaching content in the direction of more cir- 
cuit analysis. This was especially true of the more abstract and diffi- 
cult tests (e.g., figural systems, analogies, and absurdities). 
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Cost/ Effectiveness Analysis 



The technique employed for assessing cost/ effectiveness involved 
application of the following formula: 



Cost/ Effectiveness 



DC+AC+SC ,2 1 

X (l-r)^ or TT 



N g 



where: DC = Direct costs 

AC = Administrative costs 

SC - Support costs 

N = Number of students evaluated 

n 

= Variance common to test and criterion 
S = Number of course improvement suggestions 
elicited 

This formula yields an inverse index^ i. e. , a high cost/ effective- 
ness index designates less cost/ effectiveness. 

Direct costs can be considered to include the total dollar cost of 
the necessary man hours for test construction plus the cost of materials. 
Administrative costs include facility and other indirect costs. These are 
proportional to direct costs and for the present purposes can be considered 
to equal direct costs. Support costs include such factors as support equip- 
ment, replacement, test administration time, and test scoring time. As 
direct costs, administrative costs and support costs increase the cost/effec- 
tiveness of the test decreases. The variance common to the test and the 
criterion indicates the degree of relationship between the test and the 
criterion. The number of course improvement suggestions elicited is 
a function of the testes inter correlation with other predictors, as well 
as the test's validity coefficient. The greater the amount of unique vari- 
ance in the criterion that the test accounts for, the greater the amount 
of training feedback information elicited by that test^ 

In the present analysis, the test's validity coefficient was em- 
ployed rather than the number of course improvement suggestions elicit- 
ed. As the validity coefficient increases the cost/ effectiveness value, as 
calculated, decreases. Table 40 presents "best estimates" of the yearly 
costs for each factor in the cost/ effectiveness equation for each test, in- 
cluding the usual Block IV test. We have assumed that N is a constant 
value of 300 students per year. The auxiliary measures, such as achieve- 
ment motivation and physiological arousal have not been considered, inas- 
much as thcii' costs arc minimal and because many of the variables in th{) 
cost/bcnofit equation do not apply Lo ihom, AddiUon.'i.! as.sumplions aro: 

Er|c 94 



1, each sequential test item contains 12 items 

2, 300 students will complete 12 sequential test items 

3, 150 students will complete 24 sequential test items 

4, 75 students will complete 36 sequential test items 

5, the test length of the confidence test and its variations 
is 30 test items 

6, the figural systems, absurdities, analogies, and 
schematic reading tests contain 25 items 

7, four forms of each test will be constructed 

8, floor space costs $5 a square foot per year and is 
the same for each test 

9, maintenance is $800 per year 

10, duplication costs $0,01 per page 

11, typing or test preparation costs $4 per page 

12, scoring costs and test administration costs are $6 per hour 



The results of the cost/ effectiveness analysis are presented in 
Table 41, which suggests that the confidence scoring technique and the 
signal detection modification of the confidence scoring technique were 
the most cost effective of the advanced measurement techniques. The 
schematic reading, analogies, figural systems, sequential, and Block IV 
tests appear to be moderately cost effective. The absurdities test was 
least cost effective, probably because of its low correlation with the cri- 
terion. 
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Administrative Specialist Measures 



Administrative Specialist Correlational Analysis 

As with the electronic principles course, each of the adminis* 
trative specialist advanced measurement techniques was considered 
as a predictor of school performance in a concurrent validity study. 
Also included as predictors in this portion of the analysis were the 
auxiliary measures and the scores on the Block II test in the adminis- 
trative Specialist School. Three criteria of school performance were 
used: 

1. average of course block grade with the effects of 
the Block II grades removed 

2. total self*paced hours to compute administrative 
specialist course excluding the hours taken to com- 
plete Blocks I and II 

3. average of course block grade with the effects of 
the Block II grades n^oved and normalized by the 
square root of the total self-paced hours to com- 

• plete the administrative course excluding the hours 
taken to complete Blocks I and II 

The last criterion was included to allow a measure of perform- 
ance which includes the amount learned tempered by the learning time. 
The square root function was included to reHect the negative accelerated 
characteristics usually associated with learning curves. (If the square 
root transformation were not performed most of the variance of this cri- 
terion measure would have been attributable to learning time which was 
also used as a criterion measure. ) 
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The stepwise regression (Dixon, 1965) method of analysis was 
also applied to the administrative specialist data. Prior to the com- 
pletion of the stepwise regression analysis, in order to eliminate 
decimals and negative numbers, several of the advanced measure- 
ment technique scores were either multiplied by a constant or had a 
constant added. Those measurement techniques so involved, along 
with their respective additive and/ or multiplicative constants, are 
shown below in Table 42. 



Table 42 

Additive and/ or Multiplicative Constants for Certain Admin- 
istrative Specialist Advanced Measurement Techniques 

Technique Additive Constant llu 1 1 1 p 1 1 c 1 1 1 v e Constant 

Partial Knowledge 4. 00 

Thurstone Typing Samples (all) 250. 00 
Ab^rdities 7. 00 

Sigual Detection 12. 00 10. 00 

Achievement Motivation 32. 00 



The means and standard deviations of the advanced measurement 
techniques, the auxiliary measures, the Air Force aptitude variables, 
the Block 11 score, and the criterion variables for the 31 students sam- 
pled are presented in Table 43. The intercorrelation matrix for these 
same variables and subjects is presented in Table 44. 
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Table 43 



Means and Standard Deviations of the Advanced Measurement Tech- 
niques, the Auxiliary Measures, the Air Force Aptitude Variables, 
the Block 11 Score, and the Three Criterion Variables for 31 Stu- 
dents Enrolled in the Administrative Specialist Course 



Variable 


Mean 


Standard 
D eviation 


Partial Knowledge 


31. 032 


11.563 


Signal Detection (d') 


115.484 


46. 586 


Technical Words 


18.677 


4. 700 


Typing I 


264.710 


17. 162 


Typing II 


216.645 


58.756 


Typing IH 


243.387 


26.267 


Typing IV 


241.935 


25. 857 


Absurdities 


12.258 


4.604 


Need Achievement (Civilian) 


132.484 


52.789 


Need Achievement (Air Force) 


129.484 


47. 109 


Psychophysiological Arousal 


62.226 


28. 036 


General Aptitude 


51.452 


19.925 


Mechanical Aptitude 


42.097 


20. 280 


Administrative Aptitude 


57.581 


17.023 


Electronics Aptitude 


48.065 


19.394 


Block II Examination Score 


83.355 


11.960 


Final Course Average 


79.903 


10.537 


Course Hours 


119.968 


54.682 


Course Average^ 






V Course Hours 


7.983 


2.037 
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The obtained validity coefficients for final course average and 
final average divided by the square root of hours to complete course 
are generally quite acceptable. Raw hours to complete course were 
not indicated to be a criterion which can be predicted by the advanced 
measurement techniques, the Air Force aptitude tests, the usual 
school Block II examination, or the auxiliary measures. In terms of 
the two predictable criteria, the technical words, absurdity, partial 
knowledge, and d' approaches demonstrated validity coefficients which 
ranged from , 42 to , 74, These concurrent validity coefficients are ex- 
ceptionally high for this type of situation. The Thur stone scaled typing 
tests individually showed little validity as did the measures of need 
achievement. Some promise was indicated for the psychophysiological 
arousal measure in this context. To collapse systematically the inter- 
correlational matrix on the basis of the intercorrelations among the 
predictor variables, a cluster analysis was performed. The results 
of the cluster analysis are shown in Table 45, 

The results of the cluster analysis demonstrate that the largest 
cluster consists of the four Air Force aptitude tests and the Block U 
exam score. The second cluster contains the two partial knowledge 
tests and the technical words test. The two need achievement indi ces 
comprise the third clusterrThe second," third, and fourth typing tests 
constitute the fourth cluster. The first typing test, psychophysiologi- 
cal arousal, and the absurdities tests remained independent of the four 
major clusters. 

Table 44 indicates a positive relationship (r = , 35) between 
achievement motivation (civilian) and the psychophysiological arousal 
index. The electronics data, though, indicate these variables.'^ to be in- 
versely related (r = -,44), It is noted that the mean general aptitude 
test score for the electronics sample (78, 8) was significantly higher 
than that of the administrative sample (51, 5), This suggests that ap- 
titude may moderate the relationship between arousal and achievement 
motivation. That is, physiological arousal and achievement seem in- 
versely related in high aptitude groups and positively related in low ap- 
titude groups. 
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Table 46 presents the means and t values for the four aptitude 
tests across the electronics and administrative groups. 



Table 46 



Means and t-Ratios for the Four Air Force Aptitude Tests 
Across the Administrative and Electronics Groups 



Test 



Administrative Electronics 

Mean Mean t-Ratio 



General 
Mechanical 
Administrative 
Electronic 



51.45 
42. 10 
57.58 
48. 07 



78.78 
77.07 
71. 10 
86.34 



6.65* 
8.26* 
3.22* 
10.51* 



*p < .01 

Table 46 indicates a substantial difference in aptitude across 
the administrative and electronics samples. 

The first 12 steps of the stepwise regression procedure for each 
of the criteria (Dixon, 1965) are ^hown in Appendices B, C, and D of 
this report. Steps 1, 4, and 8 are presented for each of the criteria in 
Tables 47-49. 
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Selection of the most useful prediction schenie from among 
those presented in Tables 47-49 rests with the goals of the user. If 
final course average prediction represents the goal of the user, then 
the equations shown in Table 47 would be used. If, on the other hand, 
time to complete the course (with a passing grade) is the goal, then 
the equations presented in Table 48 would be used. If prediction of 
time to complete the course in conjunction with grade represents the 
user's goal, then the equations given in Table 49 would be used. 

Several of the predictors appear in all three criterion situations. 
These predictors are technical words^ absurdities, and Block II exam 
score. Technical words and absurdities were two of the advanced meas- 
urement techniques. Six other predictors appeared in two of the three 
criterion situations. These were:(a) typing I, (b) typing II, (c) typing IV, 
(d) psychophysiological arousal, (e) mechanical aptitude, and (f) Block II 
examination score. 

We note the absence of the administrative aptitude test in any of 
the regression equations. 

Summaries of analyses of variance for the significance of linear 
prediction for steps 1, 4, and 8 of each prediction scheme are shown 
in Tables 50-52, Tables 50-52 indicate statistically significant F tests 
in all instances. 
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Table 50 



Analysis of Variance Summary Tor the Significance of Linear Prediction 
for Steps One, Four, and Eight of the Stepwise Regression Procedure 
Predicting Final Course Average 

Analysis of Variance 



Step 




df 


Sum of Squares 


Mean Square 


F 


1 


Regression 


1 


1433. 933 


1433.933 


21.924* 




Residual 


29 


1896. 777 


65.406 




4 


Regression 


4 


2411. 525 


602.881 


17.053* 




Residual 


26 


919. 185 


35. 353 




8 


Regression 


8 


2693. 012 


336.626 


11.613* 




Residual 


22 


637.698 


28.986 





*p <.01. 



Table 5 1 

Analysis of Variance Summary for the Significance of Linear Prediction 
for Steps One, Four, and Eight of the Stepwise Regression Procedure 
Predicting Course Hours 

Analysis of Variance 



Step 




df 


Sum of Squares 


Mean Square 


F 


1 


Regression 


1 


41054.737 


41054.737 


24. 473* 




Residual 


29 


48648. 230 


1677.525 




4 


Regression 


4 


49210.313 


12302.578 


7. 899* 




Residual 


26 


40492.655 


1557.410 




8 


Regression 


8 


57151. 536 


7143.942 


4.828* 




Residual 


22 


32551. 432 


1479.611 





*p < .01. 
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Table 52 

Analysis of Variance Summary for the Significance of Linear Prediction 
for Steps One^ Four, and Eight of the Stepwise Regression Procedure Pre* 
dieting Final Course Average Divided by the Square Root of Course Hours 

Analysis of Variance 



step 




df 


Sum of Squares 


Mean Square 


F 


1 


Regression 


1 


67238. 834 


67238. 834 


34. 093« 




Residual 


29 


57194. 841 


1972.236 




4 


Regression 


4 


83598. 072 


20899.518 


13.302* 




Residual 


26 


40835.604 


1570. 600 




8 


Regression 


8 


90257.033 


11282. 129 


7.262* 




Residual 


22 


34176.643 


1553. 484 





* p < .01. 



Signal Detection 

In the administrative specialist phase of the correlational analy- 
sis, as with the electronic principles analysis, the signal detection meth 
od of scoring the partial knowledge test did not seem to yield a validity 
coefficient that is appreciably different from the partial knowledge test 
score. 

A receiver operator characteristic (ROC) curve depicts d* as a 
function of hit rate (HR) and false alarm rate (FAR). In the present 
context, HR is defined as the proportion of time the subject correctly 
identifies signal when presented with signal plus noise. FAR is defined 
as the proportion of time the subject incorrectly identifies noise as sig* 
nal when presented with noise. The BOC curve shown in Figure 26 pre- 
sents the average d' for subjects scoring in the top, middle, and bottom 
thirds on the final course average criterion. There is considerable dif- 
ference between the ROC curves for the lowest criterion group and the 
other two groups. 
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FALSE ALARM RATE 

Figure 26. Detection sensitivity (d*) for students scoring in 

the bottom, middle, and top thirds of the criterion. 
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Reliability Analysis 



As was the case with the electronic principles test, the Kuder* 
Richardson 20 internal consistency reliability formula was applied in 
all appropriate cases to each of the administrative specialist tests. 
Only the obtained reliability coefficient for the absurdity test was cor- 
rected through the Spearman-Brown formula. The remaining tests, 
partial knowledge and technical words, met the 25 test item require- 
ment. Table 53 presents the obtained reliability coefficients for the 
tests taken by the administrative specialist students. Internal con- 
sistency reliabilities for the four typing tests could not be calculated 
given the nature of the typing data. Instead, the number of reversals 
from the Thurstone scale was considered to be our index of typing in- 
consistency. The number of reversals was divided by the total num- 
ber of possible reversals. This proportion was subtracted from one 
to give an index of reliability. 

The data included in Table 53 indicate the partial knowledge 
and the typing scores to possess unacceptable reliability, while the tech 
nical words and absurdity tests yielded reliability indices in the accept- 
able range. 



Table 53 



Reliability Coefficients for Three Novel 
Administrative Specialist Measures 



Test 



n 



Coefficient 



Partial knowledge 
Technical words 
Absurdities 
Typing 



37 
38 
31 
38 



.50 
.77 
.77 
.54 
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Questionnaire Analysis 



After testing was completed, each administrative student com- 
pleted a structured questionnaire concerned with the desirability or un- 
desirability of each of the advanced measurement techniques. The 12 
items included in the questionnaire inquired into the examinees' opin- 
ion regarding the relevance, fairness, preferability, complexity, and 
interest of the advanced measurement techniques as compared with the 
usual multiple choice school tests. In the questionnaire, the student 
was asked to indicate his agreement or disagreement with each state- 
ment along a six point scale. The response means were then calculat- 
ed for all students. Table 53 presents the mean value, topic, and ques- 
tion number for the aforementioned questionnaire data. All scales that 
were presented to the students in the reverse direction were again re- 
versed for ease of comparison and consistency. In addition, the ques- 
tions have been regrouped according to test or test type. As was the 
case with the electronic tests, values between 3. 00 and 4. 00 on the 
scale indicate neutrality of opinion regarding the new advanced measure- 
ment techniques. Values greater than 4. 00 indicate that the subjects 
responded positively tci^|he advanced measurement techniques over the 
usual school tests, while values less than 3. 00 indicate that the sub- 
jects responded unfavorably to the advanced measurement techniques. 

Several conclusions can be derived from the data presented in 
Table 54: 

1. multiple choice tests were preferred over absurdities 
tests 

2. the advanced measurement technique test directions 
were easy to understand 

3. the advanced measurement technique experimental 
program was thought to interfere with the students' 
schedule 
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Table 54 



Mean Response to Questionnaire Items for Various Tests 




Question 
Number 


Test 


Topic 


Mean 


1 


Partial knowledge 


Pref e re nee 


3.35 


2 


Absurdities 


Preference 


1.29 


3 


Technical words 


Relevance 


4.48 


4 


Partial knowledge 


Preference 


1. 26 


5 


Absurdities 


Interest 


3. 45 


6 


Technical words 


Relevance 


3. 52 


7 


All Advanced Measurement 
Techniques 


Interest 


3. 35 


8 


All Advanced Measurement 
Techniques 


Difficulty 


3. 94 


9 


All Advanced Measurement 
Test Directions 


Difficulty 


4. 10 


10 


All Advanced Measurement 
Techniques 


Interference 


1.90 


11 


All Advanced Measurement 
Techniques 


Interest 


3. 10 


12 


All Advanced Measurement 
Techniques 


Relevance 


3. 84 
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Again^ it seems as though the more novel approaches to test 
development (e.g., absurdities, partial knowledge) were less pre- 
fered than the more traditional approaches. Also, there appeared to 
be a tendency for the students to acquiesce when responding to some 
of the questions. For instance, questions 1 and 5, which were phrased 
positively toward the advanced measurement techniques, received neu- 
tral responses. Conversely, questions 2 and 4, which were phrased 
negatively toward the advanced measurement techniques, elicited un- 
favorable responses. In effect, it seems that the items partially per- 
suaded the students to agree regardless of the attitude toward the ad- 
vanced measurement techniques. Finally, some of the students may 
have responded negatively toward the advanced measurement tech- 
niques because of the interference of the testing programs with their 
schedule. 

Interview Analysis 

Ten instructors from the administrative specialist school were 
given individual briefings on the development, rationale, and use of 
each advanced measurement technique. Each instructor was then sys- 
tematically interviewed regarding his assessment of: (1) the effective- 
ness of the new technique when compared with the old, (2) the utility of 
each technique relative to assessing student attainment of training ob- 
jectives, (3) administrative and logistic problems relative to each tech- 
nique, and (4) the possible effects of each technique on the role of the 
instructor and training evaluation specialist. Tables 55-58 present the 
instructors' evaluation of the effectiveness of each of the new advanced 
measurement techniques when compared with the usual multiple choice 
type of test. Effectiveness was rated along a scale which ranged from 
0 to 100, with a value of 50 representing effectiveness equal to that of 
multiple choice tests. A value above 50 indicates that the instructor 
considered the advanced measurement technique to be more effective 
than the multiple choice school tests, while a value below 50 indicated 
that the instructor thought the advanced measurement technique was 
less effective than the multiple choice school tests. Evaluations of 
45 to 55 indicate a neutral judgment effectiveness when comparing the 
advanced measurement techniques to the multiple choice school tests. 



114 



Table 55 



Evaluation of Effectiveness of the Partial Knowledge ""est 
as Compared with Multiple Choice Tests by Admirastrative 
Specialist Instructors at Keesler Air Force Base 

Instructor 

Number Effectiveness Comments 

1 50 No comment 

2 20 Makes them think too much. Too used 

to putting down a right answer. Too much 
of a change 

3 75 No comment 

4 75 No comment 

5 75 No comment 

6 75 Builds his morale to know he has some- 

thing right 

7 90 You find out what the student knows, A 

better view of his understanding of the 
subject 

8 75 Anything that rewards the student for a 

partial answer motivates the student and 
will help his job 

9 75 You get a chance to tell what you do know. 

Sort of like an essay test 

10 90 The method gives man credit for what he 

doesn't know. Honesty. Eliminates guess- 
work. A valid test. 

Mean 70. 00 
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Table 56 



Evaluation of Effectiveness of the Technical Words Test as 
Compared with Multiple Choice Tests by Ten Administrative 
Specialist Instructors at Keesler Air Force Base 



Instructor 

Number Effectiveness 



Comments 



1 
2 
3 
4 
5 
6 
7 



50 
65 
50 
75 
75 
50 
50 



No comment 
They'll learn more 
No comment 
No comment 
No comment 
About the same 
Same 



8 

9 

10 



85 

50 
92 



A lot don't know termi- 
nology 

Same as school tests 

Better, because if a man 
knows a terrh he knows 
something about it. You're 
pulling out what the man 
knows 



Mean 



64. 20 
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Table 57 



Evaluation of Effectiveness of the Thurstone Typing Sample as 
Compared with the Usual School Tests by Ten Administrative 
Specialist Instructors at.Keesler Air Force Base 



Instructor 



Number Effectiveness Comments 

1 90 No comment 

2 10 You can tell a bad from a good typist without it. 

3 75 No comment 

4 100 You^re able to analyze the student and what 

his problems are. 

5 75 No comment 

6 50 About the same. ^ 

7 60 A little more effective. We already have a tool 

for diagnosing typing problems. Diatype can 
be used for this purpose. 

8 95 There's no consistency in diagnosing what each 

individual thinks of a student. We need to 
reduce the causes of difficulty. 

9 75 You could measure the point to where some- 

one is efficient. He can do it this far. 

10 75 Better than what we've got. 

Mean 70.50 
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Table 58 



Evaluation of Effectiveness of the Absurdities Test as Compared 
with Multiple Choice Tests by Ten Administrative Specialist 
Instructors at Keesler Air Force Base 



Instructor 



Number Effectiveness Comments 

1 60 No comment 

2 45 It would have to be basic. 

3 65 We've had it before. 

4 75 No comment 

5 25 Wouldn't give him knowledge he needs to 

do it. 

6 60 No comment 

7 25 The student might be able to know what's 

right and what's wrong, but not be able 
to pick it out of a picture. He may over- 
look it. Too much room for error. 

8 50 About the same. 

9 0 You know material and still not be able to 

pick out what's wrong. 

10 100 With a picture you can see what's going on. 

It's realistic. . 

Mean 50. 50 
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The interview data presented in Tables 55-58 indicate that the 
instructors had a favorable opinion of the partial knowledge test, the 
technical words test, and the Thur stone typing samples. The instruc- 
tors had a neutral opinion of the absurdities test. 

Table 59 presents the instructors' estimation of the probability 
that administrative or logistics problems would arise in the use of the 
advanced measurement techniques. The instructors were asked to esti- 
mate the chances in 100 that administrative and logistics problems 
woxild arise. Accordingly, the higher the mean value shown in Table 59, 
the greater the instructors' estimation that administrative and logistics 
problems would arise. 



Table 59 

Estimation by Ten Administrative Instructors that Administrative 
or Logistics Problems Would Arise in the Use of the Advanced 
Measurement Techniques 



T echnique Probability 



Partial knowledge . 2 1 

Technical words . 15 

Thur stone typing samples , 15 

Absurdities , 23 

Traditional multiple choice tests , 14 



Table 59 indicates rather conclusively that the administrative 
instructors considered the new tests to involve very few administrative 
and logistics problems. 

The final interview question concerned the possible effects of 
the advanced measurement techniques on the role of the instructor and 
the training evaluation specialist. The most consistent effect, as seen 
by the instructors, was a need for more time when dealing with the ad- 
vanced measurement techniques, A few of the instructors said that 
their own knowledge would have to be increased in order to use the ad- 
vanced measurement tests. 



ERIC 



119 

) 



Cost/ Effectiveness Analysis 



The same cost/ effectiveness formula and definitions as employed 
for the electronic principles cost/ effectiveness analysis were used for the 
cost/ effectiveness analysis of the administrative tests. 

Table 60 presents the employed estimate of the yearly costs for 
each factor in the cost/ effectiveness equation for each test including the 
Block II test. The achievement motivation and physiological arousal 
measures were not considered, since their costs are negligible, and be- 
cause many of the variables in the cost/ effectiveness equation do not ap- 
ply to them. In computing these cost/ effectiveness indices: 

1, the validity coefficients of final course average and 
course hours were the only ones used since they are 
essentially uncorr elated while the final average divided 
by the square root of course hours is correlated with 
each of the other two criteria 

2, the validity coefficients relating the highest typing score 
obtained with the criteria were used in computing cost/ 
effectiveness of the typing tests. 
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Additionally, the following assumptions were made: 

1. 300 students will complete the administrative coarse 
in one year 

2. 4 Thurstone typing samples take 10 minutes to score 

3. each test, except for the typing tests and the Block 11 
examination, contain 25 items 

4. the Block II examination contains 50 items 

5. each test, except for the typing tests, has four examinations 

6. floor space cost is $5 a square foot per year and is the 
same for each test 

7. use of 50 typewriters, for the typing tests, for one year 
costs $300 

8. paper costs $0.01 per sheet and a pencil costs $0.05 

9. maintenance is $800 per year 

10. duplication costs $0.01 per page 

11. typing or test preparation costs $4 per page 

12. scoring costs and test administration costs are $6 per 
hour 

The cost/benefit indices for the various tests are presented in 
Tabl6 60. In Table 60, low indices indicate a relatively greater cost/ 
benefit. 

Table 60 demonstrates that the partial knowledge test, the signal 
detection modification of the partial knowledge test, and the technical 
words test were the most consistently cost/ effective of the administrative 
advanced measurement techniques. The Thurstone typing test, the ab- 
surdities test, and the Block 11 examination were only moderately cost 
effective. 
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ly. DISCUSSION, SUMMARY, AND CONCLUSIONS 



The present study possessed two interrelated but separate pur- 
poses. The first purpose involved exploring methods for conerently 
integrating the various steps in the Instructional System Development 
(ISD) cycle of the Air Force. The second purpose involved an investi- 
gation of the potential of advanced testing concepts in the Air Force 
technical training context. 

ISD Integration 

The ISD cycle involves a structured set of steps which are in- 
volved in the development of each Air Force technical training course. 
These steps are: 

1. analysis of system requirements 

2. definition of educational or training requirements 

3. development of objectives and tests 

4. planning, development, and validation of instruction 

5. conducting and evaluating instruction 

Working with available job analytic data, multidimensional scal- 
ing analysis and cluster analysis were demonstrated to provide compar- 
able results in terms of organizing job task statements into coherent en- 
tities. It seems that course training requirements and objectives can 
similarly be derived in terms of these entities (factors or clusters). The 
present work demonstrated the applicability of these entities to test devel- 
opment. If course content were organized around these entities and post- 
training evaluations were similarly rooted, the needed integrating core 
would be provided throughout the entire ISD cycle. Certainly, from the 
points of view included in the present study, there is little to preclude the 
use of the multidimensional scaling or the cluster analytic methods for 
providing the required integrating nexus. Unless such a common core is 
provided throughout, ambiguities in and misinterpretations of the job anal- 
ysis can tend to introduce early error that becomes amplified in the sys- 
tem. Moreover, the factors or clusters provide a common definitional 
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substrate for each ISD step and as such can help to minimize mismatches 
at the various nodal points. Without such a common factor approach, it 
seems apparent that the persons responsible for each stage of course de- 
velopment through the ISD system will develop different goals and defini- 
tions. The result is apt to be a number of different perceptions of the 
entity. In this case, a disorganized end result is apt to result. 

The present work also demonstrated that measurement methods, 
based on the derived factors, can yield valid, reliable, and cost/ effective 
student achievement scores. Implementation of the use of certain of the 
advanced measurement techniques might involve certain qualitative admin- 
istrative problems. However, these disadvantages were not reflected In 
the cost/ benefit analyses which, in general, supported the advanced con- 
cepts. 

The use of all of the advanced measurement techniques here inves- 
tigated cannot be defended. However, for the electronics tests, the data 
suggest the following to warrant consideration; sequential testing, figural 
systems, confidence testing, and d'. For the administrative specialist 
training consideration seems warranted of: technical words, absurdities, 
partial knowledge, and d'. 

It was thought that the advanced measurement methods, along 
with the aptitude variables and the auxiliary measures could be used as 
predictors of final course average. We have already indicated that the 
end of course criterion is somewhat contaminated since the block tests 
and the Air Force Aptitude measures involve the same intellectual and 
test taking abilities that are found in the criterion measure. There is a 
considerable proportion of variance common to the Air Force tests and 
the criterion measure that can be attributed to factors other than aptitude 
or course knowledge (e.g., reading ability, cognitive style, prior testing 
experience, etc.). Followup studies, using an on-the-job performance 
criterion would constitute a more defensible validational paradigm. 

We hypothesized earlier that most of the advanced measurement 
techniques employed were involved with unique cognitive functions not 
found in the aptitude variables or the block tests. In fact, one of two 
sets of results seem to support this hypothesis. First, the cluster anal- 
ysis demonstrated that the Air Force Aptitude tests and the block tests 
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tend to fall into the same cluster. In addition, those advanced meas- 
urement techniques using course items (e.g., sequential tests, con- 
fidence tests) also seem to be highly correlated. Second, in the re- 
gression analysis itself, fewer of the highly correlated Air Force vari- 
ables entered into the regression equation than would be expected con- 
sidering their correlations with the criterion. 

Of the six electronic principles predictors in the eighth regres- 
sion step (it will be recalled that the computer program eliminated one 
variable from the regression equation at the eighth step), four were ad- 
vanced measurement techniques --sequential test level, schematic read- 
ing/component identification, signal detection score on the confidence 
test, and civilian need achievement. The two Air Force predictors in 
this step were the electronics aptitude and the Block IV test score. 
These results also indicate the utility of the advanced measurement 
techniques for predicting final average in the electronic principles 
course. 

When predicting final course average for the administrative spe- 
cialist course, six of the predictors in the eight variable equation were 
part of the advanced measurement battery^ The six predictors included 
were the technical words test, the absurdities test, the signal detection 
method of scoring the partial knowledge test, two of the Thurstone typ- 
ing samples, and the measure of psychophysiological arousal. The two 
Air Force predictors at the eighth regression step were mechanical ap- 
titude and the Block II examination score. These results also indicate 
that the administrative advanced measurement techniques were very 
successful predictors of final course average in the administrative 
courses. 
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Summary and Conclusions 



One purpose of the present program was to describe, on an 
introductory basis, methods for providing a unifying core within the 
ISD technique as employed within tne Air Force technical training con- 
text. A second purpose was to demonstrate the adaptability and utility 
of several advanced measurement and evaluation techniques to Air Force 
student measurement. 

Multidimensional scaling and cluster analytic techniques were 
employed to order job analytic data for two Air Force technical special- 
ties- -electronics and administrative specialist. For both specialties, 
the results from both techniques were similar. Accordiri'gly, it was held 
that either the multidimensional scaling or the clustering technique will 
provide the required definitional nexus within the ISD cycle. 

A number of advanced measurement methods and techniques were 
developed on the basis of the job factors extracted from the job analytic 
data for each specialty. 

Forty-one electronics students and 31 administrative students 
were administered the respective advanced measurenient techniques. 
The students also completed a questionnaire regarding their attitudes 
toward some of the advanced measurement techniques. In addition, 10 
instructors from each school were interviewed in order to assess their 
opinion of the new measurement methods. 

The scores derived from the advanced measurement techniques 
were analyzed in regard to validity, reliability, uniqueness, and cost/ ef- 
fectiveness. Within each specialty, several of the advanced measurement 
techniques were indicated to possess psychometric, cost/ benefit, and re- 
lated properties which support a contention favoring their adoption. For 
the electronics specialty, these included: sequential testing, figural sys- 
tems, confidence testing, and scoring on the basis of theory of signal de- 
tection (d'). For the administrative specialist, the supported advanced 
measurement techniques included; technical words, absurdity recognition, 
partial knowledge scoring, and d* scoring. At least for the two job speci- 
alties considered here, the following conclusions appear warranted: 
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The multidimensional scaling and the cluster 
analytic techniques appear to possess merit as 
methods for ordering job analytic data and for 
providing coherency within ISD application , 

Extension of student achievement paper and pen 
cil testing beyond the multiple choice format is 
warranted. 
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APPENDIX A 



The First 12 Electronics Regression Equations Showing the Increase 
in Predictive Efficiency for Each Step in the Regression Process 



One 

y' = 


step (R = . 636) 
65. 586 + . 899x, 












Two 

y' = 


steps (R = . 669) 
41. 255 + . 800x_ + . 

0 


299x 

ID 










Three 

y' = 


steps (R = . 693) 
37. 943 + . 304x_ + . 


042x_ + . 

7 


330x 

10 








Four 


steps (R = .713) 










y' = 


34. 843 + . 183x_ + . 


042x_ + . 

7 


287x 

15 


+ . 112x^g 






Five 


Steps (R = .734) 








y' = 


34.516 + .060x- + . 


210x_ + . 

0 


036x_ = 

7 


= . 221x +. 133X.- 




Six 

y' = 


Steps (R = .751) 
41.679 + .067Xg + . 


082x_ + . 


047x^ - 


• .015Xg + 


. 167x^5 + 120Xjg 




r = 


Steps (R = .762) 
40.337 + 1.051x^ + 


.066Xg - 


.OI8X5 


+ . 045x^ - 


.018Xg + . 177x^g + 


.U9Xjg 


Eight 

y' = 


Steps (R = . 762) 
40.430 + 1.041x^ + 


.066Xg + 


. 044x^ 


-.018x- + 


. 176x^5 • ^^^''le 


Nine 


Steps (R = . 773) 








y' = 


43.735 + 1.089x^ + 


. 074X2 + 


.042x^ 


-.023Xg - 


.041x^^ + . 161x^g + 


.135x^3 


Ten 


Steps (R = .778) 












y' = 


44. 031+1. 336X + .081x + . 
+ . lOOx^g + . 13f7x^g 


036x^ - 


. 023Xg - . 


047x^^+.068Xjg 





Eleven Steps (R = .783) 

y' = 48. 022 + 1. 393x. + . 082x_ + . 034x + . 040x 
+ . 069x^g + . 037^1^^ + 13&x^g 

Twelve Steps (R = .786) 

y' = 49. 070 + 1. 500Xj + . 081x, + . 032x + . 036x 
+ .051x^2 + .oejx - .OrOXjg + • l38x 



025Xg -.045x^^ 



026x„ - .047x 



11 



^ The variable nuBbers are as follows: = sequential test level, x^ =cognition of 

figural systeBs, x^ = pictorial scheaatic reading, x^ = pictorial absurdities, x^ = 

conventional test score, x^ = confidence test score, x^ r signal detection score, 

o 7 

= analogies, x^ = need achieveBent (civilian), x^^ = need achieveBent (Air Force), 
Xjl = psychophysical arousal, Xjg^ general aptitude, Xj^ = Bechanlcal aptitude, Xj^ 
iidBinKstrati vr ttptitu(h% Xj^.- elortronlcs aptitudo, x hlock IV oxun scoro. 



:Dir 
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APPENDIX B 



The First 12 Administrative Regression Equations Showing the 
Increase in Prediction of Final Course Average for Each Step 

in the Regression Process 



One 


Step (R = . 


,656) 


y = 


65.552 + . 


341x^3 


Two 


Steps (R = 


.769) 


y' = 


51. 117 + . 


997Xj + 


Three 


Steps (R = 


.824) 


K = 


48.457 + . 


778Xj + 


Four 


Steps (R = 


.851) 


y' = 


53.415 + . 


961x^ + 


Five 


Steps {R = 


.870) 


y' = 


41.616 + . 


912Xj + 


Six 


SteDS (R = 


. 883) 


~ = 


37. 196 + . 


849x, + 
1 


Seven 


Steps (R = 


. 893) 


v' — 


24 948 + 


769Xj + 


Eight 


Steps (R = 


. 899) 


y' = 

Nine 


25. 870 + . 
+ .182Xjg 

Steps (R = 


715x^ + 
.906) 


y' = 


18. 159 + . 
+ .084x^3 


744x. + 
+ . Ifflx. 


Ten 


Steps (R = 


.912) 


y' = 


12. 154 + . 
+ .071x 


674x. + 
T . 078x. 


Eleven Steps (R = 


.916) 


y = 


1.056 + .668x + . 
+ . 040x + . a62x 


Twelve Steps (R = 


.920) ► 



'13 



042x^ + . 195x 



13 



046x_ + . 150x,_ + . 20Sx 
b 16 



'16 



037x^ + . 064x, , + . 139x, . + . 227x, _ 
b 11 Id lb 



047Xg + . 062Xg + . 070x^^ + . 157x^3 +. 215x^g 



16 



13 ^ • *1^6 



052x^ + .071x„ + .031x_ + .062x,, 
bo lU 11 



^10 



16 



y' = -6. 021 + . 695x + . 750x - . 818x + . 276x - . 057x + . 051x + . 096x 

+ .043x. + .042x,^ + .oi4x,, + .014 x.. + . 232x_ 
y lU 11 16 lb 



The variable numbers arc as follows: = technical words, x^^ absurdities, 



partial 



knowledge, x^ = partial knowledge (d' ), x^ = typing I. Xg = typing II, = typing III, 

= typing IV, Xg " need achievenent (civilian). Xjo" achievenent (Air Force. Xjj = 

Dsvchophysiological arousal, ^ general aptitude. Xj^ = aechanical aptitude, x^^ = adnin- 

ratlve aptitude, Xj^ = electronic aptitude, Xjg ^ block exaii score. 



APPENDIX C 



The First 11 Administrative Regression Equations Showing the Increase 
in Prediction of Course Hours for Each Step in the Regression Process 

One Step (R = . 677) 
"y' = 266. 967 - 7. 870x^ 

Two Steps (R = . 702) 
y' = 419. 879 - 7.26Xj - .620Xg 

Three Steps (R = .713) 

y' = 452. 073 - 7. 863x^ - .763Xg + .398x^2 
Four Steps (R = .741) 

y' = 535. 339 - 7. 276x- . 8l9x_ + .707x,, - 1. 108x_ 

1 5 13 16 

Five Steps (R = . 759) 

y" = 558. 426 - 7. 828x^ - . 746Xg - .216Xg + 1.094x^g - 1. 348x^g 
Six Steps (R = . 773) 

y' = 603. 721 - 7. 541x^ - 2. 179X2 " ' ^^^^5 " * ^'^^'^Q ^- ^^^^13 ' ^^^^16 
Seven Steps (R = .785) 

y' = 647. 305 - 7. 109x, - 2. 708x^ - 1. 030x^ - . 298x„ + 1. 891x, . - . 622x, _ 
-1.188x,« 1 ^ 5 9 13 15 

lb 

Eight Steps (R = . 798) 

y' = 742. 568 - 6.712X, - 2. 832x„ - .986x. - .389x„ - .426x_ + 2.065x,_ 
-.747x, - 1.225^, 2 5 8 9 13 
15 16 
Nine Steps (R = . 807) 

y' = 822.692 - 6.941X - 3.390X - 1. 231X +. 150X - .513x - .448x 
+ 2. 259x^3 - . 86#x^g -1. 29rx^g 

Ten Steps (R = .812) 

y' = 770. 384 - 6. 860x - ^. 822x - 1. 045x + . 135x - . 553x - . 554x 
+ . 173x,« + 2.058X,, - .77^x,^ - 1.25Vr 



10 ■ " "13 "15 - — '^16 



Eleven Steps (R = . 813) 

= 790. 625 - 6. 837x - 2. 87lx - 1. 058x + . 140x - . 600x - . 588x 
+ .172X -.170x + 2. 20yx -.75lx -1.^9x 



9 
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APPENDIX D 



The First 12 Administrative Regression Equations Showing the Increase 
in Prediction of Final Course Average Divided by the Square Root of 
Course Hours for Each Step in the Regression Process 

One Step (R = . 735) 
y' = 20. 342 + 3. ISSx^ 

Two Steps (R = .777) 
y' = -9. 721 + 2. 72lXj + .465Xjg 

Three Steps (R = . 801) 

y' = -15.548 + 2. 535Xj + . 146Xjj + .467Xjg 
Four Steps (R = . 820) 

y' = -86. 890 + 2. 342Xj + . 214Xg + . 153Xj ^ + . 444Xjg 
Five Steps (R = . 831) 

y' = -76.619 + 2. 559x, + .277x^ - .056x^ + . 129x_ + .472x,_ 

1 5 D 11 lo 

Six Steps (R = . 842) 

y' = -84. 430 + 2. 403x, -r . 714x„ + . 321x^ - . 067x. + . 130x, , + . 38Sx.„ 

125d111c> 

Seven Steps (R = . 849) 

y* = -91. 569 + 2. 495x, + . 850x_ + . 342x, - . 07 Ix- + . 134x, , - . 132x, „ 
+ .453Xjg 1 2 5 6 11 12 

Eight Steps (R = . 852) 

y' = -95.736 + 2. 524x. + .9l0x_ + .339x- - .069x„ + .033x,_ + . 127x., 
-. 141x^3.. 45lij^ 

Nine Steps (R = . 856) 

y • = - 102. 033 + 2. 486x, + . 885x- + . 352x- - . 064x + . 044x- _ + . 140x . 
-.245Xj2 + .180Xj^^ + .398^jg 5 6 10 11 

Ten Steps <R = .858) 

y" = -130. 780 + 2.526X + 1. 104x + .426x - .067x + .079x + . 154x 
- .228Xj2 - .178xjg +.280Xj^ + .419Xjg 

Eleven Steps (R = .861) 

y' = -162. 186 + 2.535X. + 1.274x + .475x - . lOlx- + . 093x + .096x 
+ . 132Xj J - . 197Xj2 - . 256x + . 27 Ix^^ + . 447Xjg 

Twelve Steps (R = . 863) 

y' - -171.271 + 2. 487x + 1.263X + .481x -. 103x + .077X + .046x 
+ .lOOx,,. + .143x,\ - .182x,„ - . 256^,, + .272*x,^ + .443x,_ 
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APPENDIX E 
ELECTRONICS PRINCIPLES QUESTIONNAIRE 



Name (print) Date 

Directions 

Please indicate the extent to which you agree or disagree v/ith each 
of the statements included in this booklet. The statements refer to the tenlr^ 
you have taken over the past few days. Indicate whether you: stroagly agree 
{SA), agree (A), mildly agree (MA), mildly disaigree (MD), disagree <D), or 
strongly disagree (SD) with each statement by circling the option wliich most 
closely reflects your true feeling. Please do not leave any staterAients un- 
answered* 

CIRCLE YO UR ANSWER 

1. The multiple choice test which 
allows one to assign confidence 
levels is fairer to the student 
than traditional multiple choice 
tests. (Circle one) SA 

2* Traditional multiple choice ques- 
tions test the electronic/ electri- 
cal concepts of this course better 
than the kind of questions where 
you had to complete an analogy . SA 

3. Tests like the component iden- 
tification test are more relevant 
to electronic/ electrical work 
than most multiple choice tests. 
(Cii cle one) SA 

4. The tests that we took the past few 
days were more interesting to take 
than the usual tests in this school. 
(Circle one) SA 

5 I prefer multiple choice test ques- 
tions over the kind of questions in 
the absurdities test. (Circle one) SA 



A MA MD D SD 



A MA MD D SD 



A MA MD D SD 



A MA MD D SD 



A MA ME* D &D 
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6. I would rather take a plain 
multiple choice test than a test in 
which I must assign confidence le- 
vels to my answers. (Circle one) SA A MA MD D SD 

7. I would prefer to take more analo- 
gies tests and fewer of the usual 

type of school tests. (Circle one) SA A MA MD D SD 

8. I would like to avoid taking any 
more component identification 

tests in the future. (Circle one) SA A MA MD D SD 

9. I did not like the tests we took the 
past few days because the items 

were too difficult. (Circle one) SA A MA MD D SD 

10. The test in which we had to draw 
lines connecting various compo- 
nents and parts is preferable to 
most multiple choice tests, (Circle 

one) SA A MA MD D SD 

11. I did not like the tests we took the ' 
past few days because I could not 
understand the directions. (Circle 

one) SA A MA MD D SD 

12. I did not like the tests we took the 
past few days because they inter- 
fered with my time schedule and 

studies. (Circle one) SA A MA MD D SD 

13.1 would never again like to take a 
test like the one in which we had to 
draw lines connecting various com- 
ponents and parts. (Circle one) SA A MA MD D SD 

14. I wouldn't mind taking part in any 
future testing programs like the 
one we had the past few days. 

(Circle one) SA A MA MD D SD 

15. I did not like the tests we took the 
past few days because the questions 

were too complicated. (Circle one) SA A MA MD D SD 

16. The iests we took the past few days 
Q im to measure what was taught in 

ERJC course.' "(Circle one) SA A MA MD D SD 



APPENDIX F 
ADMINISTRATIVE SPECIALIST QUESTIONNAIRE 



Name (print) Date 

Directions 

Please indicate the extent to which you agree or disagree with each 
of :he following statements included in this booklet. The statements refer 
to the tests you have taken over the past few days. Indicate whether you 
strongly agree (SA), agree (A), mildly agree (MA), mildly disagree (MD), 
disagree (D), or strongly disagree (SD) with each statement by circling the 
option which most closely reflects your true feeling. Please do not leave 
any statements unanswered. 

CIRCLE YOUR ANSWER 



1. I prefer a multiple choice test in 
which I eliminate the wrong an- 
swers rather than pick the right 
answers. (Circle one) 

2. I prefer a multiple choice test 
over a test like the absurdities 
test. (Circle one) 

3. I think tests like the technical 
words test are relevant to work- 
ing as an administrative special- 
ist. (Circle one) 

4. 1 would prefer to take a usual 
multiple choice test over a test 
in which I have to eliminate all 
the wrong answers. (Circle one) 

5. I think the absurdities test was 
most interesting and preferable 
t3 multiple choice tests. (Circle 
c ne) 

6. 1 don't think the technical words 
i 3st was very useful and I would 
r ot like to take any more tests 

1 ke that in the future. (Circle 
Q one) 

ERLC 



SA A MA MD D SB 



SA A MA MD D SD 



SA A MA MD D SD 



SA A MA MD D SD 



SA A MA MD D SD 



SA A MA MD D SD 
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7. The tests that we took the past few 
days were more interesting than 
the usual school tests. (Circle 

one) SA A MA MD D SD 

8. I did not like the tests we took 
the past few days because the 
items were too difficult. (Circle 

one) SA A MA MD D SD 

9. I did not like the tests we took the 
past few days because I could not 
understand the directions. (Circle 

one) SA A MA MD D SD 

10. I did not like the tests we 
took the past few days because 
they interfered with my time 

schedule and studies. (Circle one) SA A MA MD D SD 

11. I wouldn't mind taking part in any 
future testing programs like the 
one we had the past few days. 

(Circle one) SA A MA MD D SD 

12. I did not like the tests we took 
the past few days because they 
didn*t measure what is taught in 

the course. (Circle one) SA A MA MD D SD 
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